Jul 02 19:53:39.399872 ip-10-0-133-137 systemd[1]: Starting Kubernetes Kubelet... Jul 02 19:53:39.708418 ip-10-0-133-137 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:53:39.708418 ip-10-0-133-137 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 19:53:39.708418 ip-10-0-133-137 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:53:39.708418 ip-10-0-133-137 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 19:53:39.708418 ip-10-0-133-137 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:53:39.709804 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.709726 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 19:53:39.718133 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718110 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:53:39.718133 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718129 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:53:39.718133 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718134 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:53:39.718133 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718137 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:53:39.718133 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718141 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718145 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718148 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718151 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718154 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718157 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718160 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718166 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718169 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718171 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718174 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718177 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718182 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718186 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718190 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718193 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718196 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718199 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718201 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:53:39.718318 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718207 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718210 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718213 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718216 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718219 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718222 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718224 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718227 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718230 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718232 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718235 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718238 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718243 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718246 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718249 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718253 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718256 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718259 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718261 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:53:39.718807 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718265 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718267 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718270 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718273 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718275 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718278 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718283 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718286 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718289 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718291 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718294 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718297 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718299 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718302 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718304 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718307 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718309 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718312 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718314 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718321 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:53:39.719253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718324 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718328 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718330 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718333 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718336 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718339 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718342 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718345 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718347 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718351 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718356 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718358 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718361 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718363 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718366 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718368 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718371 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718374 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718376 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718379 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:53:39.719746 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718381 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718384 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718387 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718391 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718394 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718397 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718399 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718402 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718405 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718407 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718410 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718413 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718417 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718420 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718422 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718425 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.718430 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719542 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719549 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:53:39.720228 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719553 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719556 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719559 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719563 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719566 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719569 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719571 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719574 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719577 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719582 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719585 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719589 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719593 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719612 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719617 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719620 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719623 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719626 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719629 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719632 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:53:39.720703 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719635 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719641 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719644 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719647 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719650 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719652 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719655 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719657 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719660 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719663 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719665 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719668 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719671 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719673 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719678 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719681 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719685 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719687 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719690 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719693 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:53:39.721236 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719695 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719698 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719701 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719703 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719706 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719709 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719714 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719718 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719720 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719723 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719726 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719729 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719732 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719734 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719737 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719740 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719742 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719745 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719748 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:53:39.721732 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719753 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719756 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719759 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719761 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719764 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719767 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719770 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719772 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719775 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719779 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719783 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719789 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719792 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719795 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719798 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719801 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719804 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719807 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719809 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:53:39.722169 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719813 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719816 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719819 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719825 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719827 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719830 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719833 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719835 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719838 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719840 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719843 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719845 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719848 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719850 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719853 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719856 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719861 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719863 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.719866 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720547 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:53:39.722637 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720553 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720556 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720558 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720567 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720573 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720576 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720579 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720582 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720586 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720589 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720593 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720612 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720618 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720621 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720624 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720627 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720632 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720635 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720638 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:53:39.723114 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720640 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720643 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720645 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720648 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720651 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720654 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720656 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720659 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720661 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720664 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720669 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720672 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720675 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720677 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720680 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720683 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720685 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720695 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720698 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:53:39.723568 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720701 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720704 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720707 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720710 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720715 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720717 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720720 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720723 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720725 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720728 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720730 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720733 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720735 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720738 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720742 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720744 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720750 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720752 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720755 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720758 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:53:39.724039 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720760 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720763 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720765 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720768 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720770 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720773 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720776 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720778 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720780 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720786 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720788 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720806 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720809 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720850 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720984 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720991 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.720996 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721001 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721005 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721007 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:53:39.724499 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721010 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721013 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721016 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721019 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721022 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721024 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721030 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721033 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721035 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721038 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721041 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721044 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721046 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721051 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721055 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721058 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721061 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721064 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721067 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721070 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:53:39.725140 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721169 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721182 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721207 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721216 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721224 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721229 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721234 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721239 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721243 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721246 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721249 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721253 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721256 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721259 2564 flags.go:64] FLAG: --cgroup-root="" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721262 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721265 2564 flags.go:64] FLAG: --client-ca-file="" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721267 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721270 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721275 2564 flags.go:64] FLAG: --cluster-domain="" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721278 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721281 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721284 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721287 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721291 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 19:53:39.725743 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721294 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721297 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721300 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721303 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721306 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721309 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721312 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721315 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721320 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721323 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721326 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721329 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721332 2564 flags.go:64] FLAG: --enable-server="true" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721334 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721339 2564 flags.go:64] FLAG: --event-burst="100" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721343 2564 flags.go:64] FLAG: --event-qps="50" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721346 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721350 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721352 2564 flags.go:64] FLAG: --eviction-hard="" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721356 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721359 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721362 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721365 2564 flags.go:64] FLAG: --eviction-soft="" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721368 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721370 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 19:53:39.726581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721375 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721377 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721380 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721383 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721386 2564 flags.go:64] FLAG: --feature-gates="" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721390 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721393 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721395 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721398 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721402 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721404 2564 flags.go:64] FLAG: --help="false" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721407 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-133-137.ec2.internal" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721410 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721413 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721415 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721419 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721424 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721427 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721429 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721432 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721435 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721439 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721442 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721445 2564 flags.go:64] FLAG: --kube-reserved="" Jul 02 19:53:39.727626 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721448 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721451 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721454 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721457 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721460 2564 flags.go:64] FLAG: --lock-file="" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721463 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721465 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721468 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721474 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721477 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721480 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721483 2564 flags.go:64] FLAG: --logging-format="text" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721486 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721489 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721492 2564 flags.go:64] FLAG: --manifest-url="" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721494 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721499 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721502 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721506 2564 flags.go:64] FLAG: --max-pods="110" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721509 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721512 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721514 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721518 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721521 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721524 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 19:53:39.728627 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721527 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721535 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721538 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721541 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721547 2564 flags.go:64] FLAG: --pod-cidr="" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721550 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721555 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721559 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721562 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721565 2564 flags.go:64] FLAG: --port="10250" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721568 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721570 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e962966ca91ab4bb" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721574 2564 flags.go:64] FLAG: --qos-reserved="" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721577 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721579 2564 flags.go:64] FLAG: --register-node="true" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721582 2564 flags.go:64] FLAG: --register-with-taints="" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721586 2564 flags.go:64] FLAG: --registry-burst="10" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721590 2564 flags.go:64] FLAG: --registry-qps="5" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721593 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721596 2564 flags.go:64] FLAG: --reserved-memory="" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721610 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721628 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721639 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721643 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721646 2564 flags.go:64] FLAG: --runonce="false" Jul 02 19:53:39.729294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721650 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721653 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721657 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721660 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721663 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721666 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721669 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721676 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721679 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721682 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721685 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721688 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721691 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721694 2564 flags.go:64] FLAG: --system-cgroups="" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721697 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721705 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721708 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721711 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721717 2564 flags.go:64] FLAG: --tls-min-version="" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721720 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721722 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721725 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721729 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721732 2564 flags.go:64] FLAG: --v="2" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721737 2564 flags.go:64] FLAG: --version="false" Jul 02 19:53:39.729928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721742 2564 flags.go:64] FLAG: --vmodule="" Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721746 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.721750 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721888 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721893 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721897 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721900 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721903 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721905 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721908 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721910 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721913 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721915 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721919 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721924 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721928 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721931 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721934 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721937 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:53:39.730526 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721940 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721944 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721948 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721951 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721954 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721956 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721959 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721962 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721964 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721967 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721969 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721972 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721975 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721978 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721981 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721984 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721988 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721990 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721993 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:53:39.731033 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721995 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.721998 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722001 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722004 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722006 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722009 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722011 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722014 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722017 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722020 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722023 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722025 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722028 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722030 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722033 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722035 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722038 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722040 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722043 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722046 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:53:39.731524 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722048 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722051 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722053 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722056 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722059 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722061 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722064 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722066 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722069 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722073 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722076 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722079 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722081 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722084 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722086 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722089 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722091 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722094 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722096 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722098 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:53:39.732158 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722102 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722105 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722107 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722110 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722112 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722115 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722118 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722122 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722125 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722128 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722130 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722133 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722136 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722139 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722141 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722144 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722146 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722149 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722151 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722154 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:53:39.732770 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722156 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722161 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722163 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.722166 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.722765 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.730445 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.730460 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.731018 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.731025 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731091 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731096 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731099 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731104 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731108 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:53:39.733253 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731112 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731114 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731117 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731120 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731123 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731127 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731130 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731133 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731137 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731140 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731144 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731147 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731149 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731152 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731155 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731158 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731161 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731163 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731166 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731170 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:53:39.733617 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731173 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731176 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731178 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731181 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731185 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731190 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731193 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731195 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731198 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731201 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731203 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731206 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731208 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731211 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731213 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731216 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731218 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731221 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731223 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731226 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:53:39.734122 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731229 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731231 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731234 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731236 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731239 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731241 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731244 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731247 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731249 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731252 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731255 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731258 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731261 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731263 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731266 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731268 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731272 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731274 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731277 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731280 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:53:39.734593 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731282 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731285 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731288 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731290 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731292 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731295 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731299 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731302 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731305 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731308 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731311 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731313 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731316 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731319 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731322 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731325 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731328 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731330 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731333 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:53:39.735099 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731335 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731338 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731340 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731343 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731346 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731349 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731352 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731355 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731357 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731360 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731363 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731365 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731367 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731370 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:53:39.735543 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731372 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.731378 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731499 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731503 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731506 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731508 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731511 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731513 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731516 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731519 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731522 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731524 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731527 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731530 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731533 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:53:39.735879 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731535 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731538 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731541 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731545 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731548 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731551 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731554 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731556 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731559 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731562 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731565 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731568 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731571 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731573 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731576 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731578 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731581 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731583 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731586 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731588 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:53:39.736246 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731591 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731593 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731611 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731616 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731619 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731623 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731625 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731628 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731631 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731634 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731636 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731639 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731641 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731644 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731647 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731649 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731652 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731654 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731657 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:53:39.736745 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731659 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731662 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731665 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731667 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731671 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731673 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731676 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731679 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731681 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731683 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731686 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731688 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731691 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731693 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731696 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731698 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731700 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731704 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731708 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731710 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:53:39.737195 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731713 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731715 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731718 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731721 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731723 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731726 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731728 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731731 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731734 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731736 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731739 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731741 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731744 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731747 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731749 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731752 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731755 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731758 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731760 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:53:39.737696 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731763 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731765 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731768 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731770 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731772 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731775 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731777 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:39.731780 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.731785 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.731891 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.733767 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.734800 2564 server.go:1019] "Starting client certificate rotation" Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.734907 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:53:39.738148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.734937 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:53:39.738461 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.736901 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:53:39.739153 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.739131 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:53:39.749014 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.748997 2564 log.go:25] "Validated CRI v1 runtime API" Jul 02 19:53:39.752996 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.752982 2564 log.go:25] "Validated CRI v1 image API" Jul 02 19:53:39.753953 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.753939 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 19:53:39.757345 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.757325 2564 fs.go:135] Filesystem UUIDs: map[3cfa315a-488a-4e34-ab2b-3fa58ec93aa3:/dev/nvme0n1p4 7448e59c-9758-4ed3-92db-f57cc8a4dded:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 02 19:53:39.757414 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.757343 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 19:53:39.763084 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.763064 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:53:39.763991 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.763886 2564 manager.go:216] Machine: {Timestamp:2026-07-02 19:53:39.762751792 +0000 UTC m=+0.268348469 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099672 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec293b2c3ce5f72fe70bfdaab76316c5 SystemUUID:ec293b2c-3ce5-f72f-e70b-fdaab76316c5 BootID:a9f429a1-dd39-4e3e-8b8e-99899da9fd76 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0c:c7:7d:89:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0c:c7:7d:89:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:01:43:95:5d:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 19:53:39.763991 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.763986 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 19:53:39.764111 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.764061 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 19:53:39.767104 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.767083 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 19:53:39.767235 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.767107 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 19:53:39.767893 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.767882 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 19:53:39.767923 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.767895 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 19:53:39.767923 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.767902 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:53:39.767923 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.767915 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:53:39.769041 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.769030 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 19:53:39.771566 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.771555 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:53:39.771689 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.771680 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 19:53:39.771734 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.771708 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 02 19:53:39.771734 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.771718 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 19:53:39.771734 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.771727 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 19:53:39.771841 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.771736 2564 kubelet.go:398] "Adding apiserver pod source" Jul 02 19:53:39.771841 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.771745 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 19:53:39.772108 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.772096 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:53:39.772154 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.772120 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:53:39.773850 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.773831 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 19:53:39.775547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.775532 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 19:53:39.775638 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.775561 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 19:53:39.776464 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776453 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 19:53:39.776512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776470 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 19:53:39.776512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776476 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 19:53:39.776512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776483 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 19:53:39.776512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776496 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 19:53:39.776512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776501 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 19:53:39.776512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776512 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 19:53:39.776718 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776517 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 19:53:39.776718 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776524 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 19:53:39.776718 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776530 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 19:53:39.776718 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776539 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 19:53:39.776718 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.776544 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 19:53:39.777700 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.777690 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 19:53:39.777700 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.777701 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 19:53:39.781084 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.781070 2564 server.go:1268] "Started kubelet" Jul 02 19:53:39.781192 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.781145 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 19:53:39.781470 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.781194 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 19:53:39.781696 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.781641 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 19:53:39.781785 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.781710 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 19:53:39.781785 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.781760 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 19:53:39.781900 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.781845 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 19:53:39.782282 ip-10-0-133-137 systemd[1]: Started Kubernetes Kubelet. Jul 02 19:53:39.783371 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.783342 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 19:53:39.784100 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.784085 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 02 19:53:39.785443 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.785421 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58n52" Jul 02 19:53:39.786296 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.786277 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 19:53:39.787965 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.787208 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-137.ec2.internal.18be911483bf9ca7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-137.ec2.internal,UID:ip-10-0-133-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-137.ec2.internal,},FirstTimestamp:2026-07-02 19:53:39.781041319 +0000 UTC m=+0.286637996,LastTimestamp:2026-07-02 19:53:39.781041319 +0000 UTC m=+0.286637996,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-137.ec2.internal,}" Jul 02 19:53:39.788480 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.788461 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 19:53:39.788579 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.788485 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 19:53:39.788579 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.788572 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:39.789019 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.788980 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 19:53:39.789019 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.788986 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 19:53:39.789019 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.789012 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 19:53:39.789212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.789160 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 02 19:53:39.789212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.789171 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 02 19:53:39.789810 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.789790 2564 factory.go:55] Registering systemd factory Jul 02 19:53:39.789906 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.789815 2564 factory.go:223] Registration of the systemd container factory successfully Jul 02 19:53:39.790111 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.790093 2564 factory.go:153] Registering CRI-O factory Jul 02 19:53:39.790111 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.790113 2564 factory.go:223] Registration of the crio container factory successfully Jul 02 19:53:39.790237 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.790169 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 19:53:39.790237 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.790211 2564 factory.go:103] Registering Raw factory Jul 02 19:53:39.790237 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.790229 2564 manager.go:1196] Started watching for new ooms in manager Jul 02 19:53:39.790702 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.790689 2564 manager.go:349] Starting recovery of all containers Jul 02 19:53:39.792409 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.792388 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58n52" Jul 02 19:53:39.794324 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.794297 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 19:53:39.794416 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.794397 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 19:53:39.802531 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.802413 2564 manager.go:354] Recovery completed Jul 02 19:53:39.806804 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.806793 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:53:39.808783 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.808765 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:53:39.808836 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.808807 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:53:39.808836 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.808824 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:53:39.809347 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.809334 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 19:53:39.809396 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.809347 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 19:53:39.809396 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.809365 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:53:39.809481 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.809347 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:53:39.810171 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.810157 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:53:39.810222 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.810182 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:53:39.810222 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.810193 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:53:39.810280 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.810250 2564 policy_none.go:49] "None policy: Start" Jul 02 19:53:39.810280 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.810263 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 19:53:39.810334 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.810283 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 19:53:39.811155 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.811144 2564 policy_none.go:47] "Start" Jul 02 19:53:39.843138 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.843121 2564 manager.go:337] "Starting Device Plugin manager" Jul 02 19:53:39.843224 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.843153 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 19:53:39.843224 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.843163 2564 server.go:87] "Starting device plugin registration server" Jul 02 19:53:39.843429 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.843414 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 19:53:39.843531 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.843432 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 19:53:39.843531 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.843486 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 19:53:39.843648 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.843581 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 19:53:39.843648 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.843591 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 19:53:39.844294 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.844269 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 19:53:39.844366 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.844327 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:39.910207 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.910164 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 19:53:39.911369 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.911355 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 19:53:39.911437 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.911391 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 19:53:39.911437 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.911414 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 19:53:39.911536 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.911446 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 19:53:39.914027 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.914006 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:53:39.943722 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.943703 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:53:39.944802 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.944786 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:53:39.944891 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.944820 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:53:39.944891 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.944834 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:53:39.944891 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.944863 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-137.ec2.internal" Jul 02 19:53:39.950233 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:39.950219 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-137.ec2.internal" Jul 02 19:53:39.950302 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.950239 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-137.ec2.internal\": node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:39.966633 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:39.966563 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:40.011902 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.011877 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"] Jul 02 19:53:40.012027 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.011942 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:53:40.012851 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.012837 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:53:40.012938 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.012867 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:53:40.012938 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.012881 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:53:40.013972 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.013957 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:53:40.014108 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014080 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:40.014151 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014125 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:53:40.014662 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014648 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:53:40.014729 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014658 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:53:40.014729 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014672 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:53:40.014729 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014679 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:53:40.014729 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014689 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:53:40.014729 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.014695 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:53:40.015613 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.015585 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" Jul 02 19:53:40.015660 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.015630 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:53:40.016286 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.016270 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:53:40.016355 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.016297 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:53:40.016355 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:40.016309 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:53:40.031135 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:40.031114 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-137.ec2.internal\" not found" node="ip-10-0-133-137.ec2.internal" Jul 02 19:53:40.036541 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:40.036524 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-137.ec2.internal\" not found" node="ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.203856 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.203836 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.228404 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.228381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/176a852dc8aa3e063105495696527f75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"176a852dc8aa3e063105495696527f75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.228512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.228407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/176a852dc8aa3e063105495696527f75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"176a852dc8aa3e063105495696527f75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.228512 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.228424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c581f67582f2b7493459f83b250c1488-config\") pod \"kube-apiserver-proxy-ip-10-0-133-137.ec2.internal\" (UID: \"c581f67582f2b7493459f83b250c1488\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.304442 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.304402 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.328776 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.328758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/176a852dc8aa3e063105495696527f75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"176a852dc8aa3e063105495696527f75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.328859 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.328784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/176a852dc8aa3e063105495696527f75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"176a852dc8aa3e063105495696527f75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.328859 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.328803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c581f67582f2b7493459f83b250c1488-config\") pod \"kube-apiserver-proxy-ip-10-0-133-137.ec2.internal\" (UID: \"c581f67582f2b7493459f83b250c1488\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.328859 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.328841 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c581f67582f2b7493459f83b250c1488-config\") pod \"kube-apiserver-proxy-ip-10-0-133-137.ec2.internal\" (UID: \"c581f67582f2b7493459f83b250c1488\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.328948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.328880 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/176a852dc8aa3e063105495696527f75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"176a852dc8aa3e063105495696527f75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.328948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.328877 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/176a852dc8aa3e063105495696527f75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"176a852dc8aa3e063105495696527f75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.405044 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.404983 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.471600 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.471577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.476201 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.476180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" Jul 02 19:53:41.505555 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.505521 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.606145 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.606103 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.706581 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.706500 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.807100 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.807068 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.871543 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.871512 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 19:53:41.872101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.871655 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:53:41.907814 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:41.907789 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:41.931900 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.931866 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 19:48:40 +0000 UTC" deadline="2028-02-29 08:41:59.389954691 +0000 UTC" Jul 02 19:53:41.931900 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:41.931899 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14556h48m17.458059223s" Jul 02 19:53:42.007125 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:42.007093 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc581f67582f2b7493459f83b250c1488.slice/crio-5a69a6ba3952bd4b4d56a31a4590cff4300d466035689691e85f4255f1f1e2eb WatchSource:0}: Error finding container 5a69a6ba3952bd4b4d56a31a4590cff4300d466035689691e85f4255f1f1e2eb: Status 404 returned error can't find the container with id 5a69a6ba3952bd4b4d56a31a4590cff4300d466035689691e85f4255f1f1e2eb Jul 02 19:53:42.007336 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:42.007317 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176a852dc8aa3e063105495696527f75.slice/crio-419dfe91abdc3bd359cdcd928701701d477fa0784529561a77f80fb5df4bb6ba WatchSource:0}: Error finding container 419dfe91abdc3bd359cdcd928701701d477fa0784529561a77f80fb5df4bb6ba: Status 404 returned error can't find the container with id 419dfe91abdc3bd359cdcd928701701d477fa0784529561a77f80fb5df4bb6ba Jul 02 19:53:42.008387 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.008372 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:42.011801 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.011786 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:53:42.051685 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.051628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" event={"ID":"176a852dc8aa3e063105495696527f75","Type":"ContainerStarted","Data":"419dfe91abdc3bd359cdcd928701701d477fa0784529561a77f80fb5df4bb6ba"} Jul 02 19:53:42.052606 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.052588 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" event={"ID":"c581f67582f2b7493459f83b250c1488","Type":"ContainerStarted","Data":"5a69a6ba3952bd4b4d56a31a4590cff4300d466035689691e85f4255f1f1e2eb"} Jul 02 19:53:42.108774 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.108746 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:42.209262 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.209234 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:42.263280 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.263215 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:53:42.310204 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.310180 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:42.357326 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.357299 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:53:42.410804 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.410778 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found" Jul 02 19:53:42.449318 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.449291 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:53:42.525998 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.525925 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" Jul 02 19:53:42.537003 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.536972 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:53:42.538120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.538097 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" Jul 02 19:53:42.546742 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.546668 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:53:42.909640 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.909572 2564 apiserver.go:52] "Watching apiserver" Jul 02 19:53:42.919455 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.919372 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 19:53:42.920363 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.920327 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-qw9rz","openshift-multus/multus-gmft6","openshift-network-diagnostics/network-check-target-8ngzh","openshift-ovn-kubernetes/ovnkube-node-xj2jm","kube-system/global-pull-secret-syncer-g7p4v","kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc","openshift-image-registry/node-ca-mbbgp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal","openshift-multus/multus-additional-cni-plugins-dgdjb","openshift-multus/network-metrics-daemon-gm4b8","openshift-network-operator/iptables-alerter-zzfph","kube-system/konnectivity-agent-wdjp6","openshift-cluster-node-tuning-operator/tuned-pb449"] Jul 02 19:53:42.921832 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.921807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.922984 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.922939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:42.923885 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.923867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:42.923972 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.923931 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:42.924308 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.924288 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4glzx\"" Jul 02 19:53:42.924522 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.924300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 19:53:42.924522 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.924487 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 19:53:42.924647 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.924614 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 19:53:42.925274 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.925256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.925844 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.925751 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 19:53:42.925991 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.925971 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 19:53:42.926065 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.926056 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-shvgj\"" Jul 02 19:53:42.926876 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.926820 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 19:53:42.928370 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.927347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:42.928370 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.927504 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:42.928370 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.928139 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 19:53:42.930610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.929660 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 19:53:42.930610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.929718 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 19:53:42.930610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.929929 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 19:53:42.930610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.930003 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:42.930610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.930129 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 19:53:42.930610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.930542 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smsxd\"" Jul 02 19:53:42.930957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.930910 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 19:53:42.930957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.930967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:42.933534 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.933186 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 19:53:42.933534 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.933354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:53:42.933680 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.933623 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 19:53:42.933738 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.933698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 19:53:42.933790 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.933734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwzh8\"" Jul 02 19:53:42.933968 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.933942 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:42.934589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.934564 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 19:53:42.934693 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.934678 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lv62z\"" Jul 02 19:53:42.935768 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.935750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gmft6" Jul 02 19:53:42.936292 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.936276 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 19:53:42.936866 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.936609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sgns2\"" Jul 02 19:53:42.936866 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.936813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 19:53:42.937200 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937105 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:42.937280 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f75a85a2-4ff5-42ec-938b-e9f3e343d2ae-konnectivity-ca\") pod \"konnectivity-agent-wdjp6\" (UID: \"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae\") " pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:42.937280 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-kubelet\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-var-lib-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-etc-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.937388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-run-netns\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-ovnkube-config\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-env-overrides\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-ovnkube-script-lib\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-device-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.937641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-sys-fs\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.937641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f75a85a2-4ff5-42ec-938b-e9f3e343d2ae-agent-certs\") pod \"konnectivity-agent-wdjp6\" (UID: \"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae\") " pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:42.937641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937659 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-node-log\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-registration-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-slash\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937781 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-ovn\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-run-ovn-kubernetes\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-cni-netd\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vzk2\" (UniqueName: \"kubernetes.io/projected/53c54e50-2fb0-439f-89a2-19719be80f90-kube-api-access-6vzk2\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48751f64-539d-4c0f-a40f-84264b0b6720-dbus\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:42.937963 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.937985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-socket-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-systemd-units\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-systemd\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938052 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-log-socket\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-cni-bin\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53c54e50-2fb0-439f-89a2-19719be80f90-ovn-node-metrics-cert\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48751f64-539d-4c0f-a40f-84264b0b6720-kubelet-config\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:42.938536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5pwg\" (UniqueName: \"kubernetes.io/projected/afe97c83-69a0-4435-a53b-2d32b8ba330b-kube-api-access-w5pwg\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:42.938987 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938595 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 19:53:42.938987 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:42.938668 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:42.938987 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938753 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 19:53:42.938987 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.938794 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cvnt7\"" Jul 02 19:53:42.939177 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.939168 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 19:53:42.940045 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.939505 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 19:53:42.940045 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.939616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:42.940045 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.939671 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8zwb2\"" Jul 02 19:53:42.940045 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.939725 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 19:53:42.941931 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.941913 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:53:42.942019 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.941951 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 19:53:42.942246 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.942225 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7ffnv\"" Jul 02 19:53:42.942872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.942854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 19:53:42.958861 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.958842 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldh8q" Jul 02 19:53:42.966244 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:42.966224 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldh8q" Jul 02 19:53:43.026738 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.026707 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 19:53:43.038393 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038368 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53c54e50-2fb0-439f-89a2-19719be80f90-ovn-node-metrics-cert\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.038536 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48751f64-539d-4c0f-a40f-84264b0b6720-kubelet-config\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:43.038630 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-lib-modules\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.038818 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-cni-bin\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.038908 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.038960 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrqk\" (UniqueName: \"kubernetes.io/projected/4c4f1949-5ea4-4839-9ccd-a3187ec13124-kube-api-access-6qrqk\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.038960 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038927 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48751f64-539d-4c0f-a40f-84264b0b6720-kubelet-config\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:43.039052 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.038954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.039052 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f75a85a2-4ff5-42ec-938b-e9f3e343d2ae-konnectivity-ca\") pod \"konnectivity-agent-wdjp6\" (UID: \"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae\") " pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:43.039052 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-var-lib-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.039181 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.039181 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039082 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.039181 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-env-overrides\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.039181 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039151 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-var-lib-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.039181 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039151 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 19:53:43.039392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039219 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.039392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-ovnkube-script-lib\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.039392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-tmp\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.039392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f313f82-0e16-4e1c-8888-727cf6ea662c-host\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.039392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5h5j\" (UniqueName: \"kubernetes.io/projected/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-kube-api-access-r5h5j\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.039392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-hostroot\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk999\" (UniqueName: \"kubernetes.io/projected/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-kube-api-access-tk999\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c4f1949-5ea4-4839-9ccd-a3187ec13124-iptables-alerter-script\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c4f1949-5ea4-4839-9ccd-a3187ec13124-host-slash\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-node-log\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-k8s-cni-cncf-io\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhg27\" (UniqueName: \"kubernetes.io/projected/d9291347-9dc5-48e1-b2af-ea6db41aa368-kube-api-access-xhg27\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-ovn\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039698 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-env-overrides\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.039713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czh92\" (UniqueName: \"kubernetes.io/projected/1f313f82-0e16-4e1c-8888-727cf6ea662c-kube-api-access-czh92\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-conf-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.039824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-socket-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040030 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-node-log\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-kubernetes\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-system-cni-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-systemd\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-log-socket\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-cni-bin\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040278 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-ovnkube-script-lib\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysconfig\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040333 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f75a85a2-4ff5-42ec-938b-e9f3e343d2ae-konnectivity-ca\") pod \"konnectivity-agent-wdjp6\" (UID: \"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae\") " pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040337 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-systemd\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040373 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-cni-bin\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040399 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-log-socket\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040442 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-socket-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.040478 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040529 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-ovn\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.040557 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret podName:48751f64-539d-4c0f-a40f-84264b0b6720 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:43.540538146 +0000 UTC m=+2.909274865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret") pod "global-pull-secret-syncer-g7p4v" (UID: "48751f64-539d-4c0f-a40f-84264b0b6720") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-tuned\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.040649 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr86l\" (UniqueName: \"kubernetes.io/projected/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-kube-api-access-jr86l\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-os-release\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pwg\" (UniqueName: \"kubernetes.io/projected/afe97c83-69a0-4435-a53b-2d32b8ba330b-kube-api-access-w5pwg\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-kubelet\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-etc-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-run\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040836 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-kubelet\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-var-lib-kubelet\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-multus-certs\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-run-netns\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-ovnkube-config\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-modprobe-d\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.040995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f313f82-0e16-4e1c-8888-727cf6ea662c-serviceca\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-cni-multus\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-etc-kubernetes\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cnibin\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.041187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-device-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-sys-fs\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f75a85a2-4ff5-42ec-938b-e9f3e343d2ae-agent-certs\") pod \"konnectivity-agent-wdjp6\" (UID: \"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae\") " pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041162 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-run-netns\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-sys\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-cnibin\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041217 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-etc-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041255 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-run-openvswitch\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-cni-binary-copy\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041415 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysctl-conf\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-hosts-file\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-tmp-dir\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-socket-dir-parent\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-registration-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.042009 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-slash\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-run-ovn-kubernetes\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-cni-netd\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vzk2\" (UniqueName: \"kubernetes.io/projected/53c54e50-2fb0-439f-89a2-19719be80f90-kube-api-access-6vzk2\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48751f64-539d-4c0f-a40f-84264b0b6720-dbus\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041712 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53c54e50-2fb0-439f-89a2-19719be80f90-ovnkube-config\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-os-release\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-kubelet\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysctl-d\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-systemd\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-host\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041859 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-device-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-cni-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-daemon-config\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041923 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-sys-fs\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-system-cni-dir\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.041981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqxk\" (UniqueName: \"kubernetes.io/projected/9e4355c3-c0aa-4c59-9614-ad46da3f012a-kube-api-access-vsqxk\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.042800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-systemd-units\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-netns\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042421 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042564 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afe97c83-69a0-4435-a53b-2d32b8ba330b-registration-dir\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042620 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-slash\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-run-ovn-kubernetes\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-host-cni-netd\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.042892 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48751f64-539d-4c0f-a40f-84264b0b6720-dbus\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.043047 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53c54e50-2fb0-439f-89a2-19719be80f90-systemd-units\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.043872 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.043049 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53c54e50-2fb0-439f-89a2-19719be80f90-ovn-node-metrics-cert\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.044912 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.044893 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f75a85a2-4ff5-42ec-938b-e9f3e343d2ae-agent-certs\") pod \"konnectivity-agent-wdjp6\" (UID: \"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae\") " pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:43.052579 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.052553 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:53:43.052579 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.052576 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:53:43.052724 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.052590 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7d874 for pod openshift-network-diagnostics/network-check-target-8ngzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:43.052724 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.052658 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874 podName:dea8b301-09ab-4de3-93fe-f4cc9c01dbab nodeName:}" failed. No retries permitted until 2026-07-02 19:53:43.552641029 +0000 UTC m=+2.921377746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7d874" (UniqueName: "kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874") pod "network-check-target-8ngzh" (UID: "dea8b301-09ab-4de3-93fe-f4cc9c01dbab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:43.053934 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.053903 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pwg\" (UniqueName: \"kubernetes.io/projected/afe97c83-69a0-4435-a53b-2d32b8ba330b-kube-api-access-w5pwg\") pod \"aws-ebs-csi-driver-node-pl2xc\" (UID: \"afe97c83-69a0-4435-a53b-2d32b8ba330b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.054789 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.054769 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vzk2\" (UniqueName: \"kubernetes.io/projected/53c54e50-2fb0-439f-89a2-19719be80f90-kube-api-access-6vzk2\") pod \"ovnkube-node-xj2jm\" (UID: \"53c54e50-2fb0-439f-89a2-19719be80f90\") " pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.092368 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.092337 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:53:43.143231 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.143231 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysconfig\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143482 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143290 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysconfig\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143482 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-tuned\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143482 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr86l\" (UniqueName: \"kubernetes.io/projected/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-kube-api-access-jr86l\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143618 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-os-release\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.143680 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143644 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-os-release\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.143680 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-run\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143786 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-var-lib-kubelet\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143786 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143699 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-run\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143786 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-multus-certs\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.143786 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-modprobe-d\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143786 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143767 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-var-lib-kubelet\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f313f82-0e16-4e1c-8888-727cf6ea662c-serviceca\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143811 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-multus-certs\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-cni-multus\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-etc-kubernetes\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143875 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-modprobe-d\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143882 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143907 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-etc-kubernetes\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cnibin\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-sys\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143948 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cnibin\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143949 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-cni-multus\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.143988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.143965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-cnibin\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-cni-binary-copy\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144006 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-sys\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysctl-conf\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-hosts-file\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144072 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-cnibin\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-tmp-dir\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-socket-dir-parent\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-os-release\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-kubelet\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144192 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-hosts-file\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144207 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysctl-conf\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144222 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f313f82-0e16-4e1c-8888-727cf6ea662c-serviceca\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysctl-d\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-systemd\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-host\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144287 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-socket-dir-parent\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.144569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-cni-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144318 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-sysctl-d\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144320 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-kubelet\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-daemon-config\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144376 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-systemd\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144380 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-os-release\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144432 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-cni-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144445 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-host\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144449 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-tmp-dir\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-system-cni-dir\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqxk\" (UniqueName: \"kubernetes.io/projected/9e4355c3-c0aa-4c59-9614-ad46da3f012a-kube-api-access-vsqxk\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144533 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-system-cni-dir\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-cni-binary-copy\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-netns\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144608 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e4355c3-c0aa-4c59-9614-ad46da3f012a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-lib-modules\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144644 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-netns\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.145450 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-cni-bin\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144693 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-var-lib-cni-bin\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrqk\" (UniqueName: \"kubernetes.io/projected/4c4f1949-5ea4-4839-9ccd-a3187ec13124-kube-api-access-6qrqk\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.144643 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144726 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-lib-modules\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-tmp\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f313f82-0e16-4e1c-8888-727cf6ea662c-host\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.144771 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs podName:d9291347-9dc5-48e1-b2af-ea6db41aa368 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:43.644751108 +0000 UTC m=+3.013487813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs") pod "network-metrics-daemon-gm4b8" (UID: "d9291347-9dc5-48e1-b2af-ea6db41aa368") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144790 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f313f82-0e16-4e1c-8888-727cf6ea662c-host\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h5j\" (UniqueName: \"kubernetes.io/projected/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-kube-api-access-r5h5j\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-hostroot\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk999\" (UniqueName: \"kubernetes.io/projected/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-kube-api-access-tk999\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c4f1949-5ea4-4839-9ccd-a3187ec13124-iptables-alerter-script\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144790 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-daemon-config\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c4f1949-5ea4-4839-9ccd-a3187ec13124-host-slash\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-k8s-cni-cncf-io\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.146328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhg27\" (UniqueName: \"kubernetes.io/projected/d9291347-9dc5-48e1-b2af-ea6db41aa368-kube-api-access-xhg27\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.144987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh92\" (UniqueName: \"kubernetes.io/projected/1f313f82-0e16-4e1c-8888-727cf6ea662c-kube-api-access-czh92\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-conf-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145073 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-kubernetes\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c4f1949-5ea4-4839-9ccd-a3187ec13124-host-slash\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-system-cni-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145131 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-kubernetes\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145071 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-host-run-k8s-cni-cncf-io\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145166 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-multus-conf-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145197 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-hostroot\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145230 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-system-cni-dir\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145500 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c4f1949-5ea4-4839-9ccd-a3187ec13124-iptables-alerter-script\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.145535 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e4355c3-c0aa-4c59-9614-ad46da3f012a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.147187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.146218 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-etc-tuned\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.147616 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.147286 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-tmp\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.151703 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.151673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr86l\" (UniqueName: \"kubernetes.io/projected/ecdef8f6-2d6a-4491-ae2e-133524b09d1e-kube-api-access-jr86l\") pod \"tuned-pb449\" (UID: \"ecdef8f6-2d6a-4491-ae2e-133524b09d1e\") " pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.153689 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.153661 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h5j\" (UniqueName: \"kubernetes.io/projected/4960089f-b7f6-4daa-ae08-c3d3df7cfe52-kube-api-access-r5h5j\") pod \"node-resolver-qw9rz\" (UID: \"4960089f-b7f6-4daa-ae08-c3d3df7cfe52\") " pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.154109 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.154070 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhg27\" (UniqueName: \"kubernetes.io/projected/d9291347-9dc5-48e1-b2af-ea6db41aa368-kube-api-access-xhg27\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:43.154528 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.154507 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk999\" (UniqueName: \"kubernetes.io/projected/24d25ab8-fc60-44d8-9a1b-e6eddd2f1741-kube-api-access-tk999\") pod \"multus-gmft6\" (UID: \"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741\") " pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.154619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.154509 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqxk\" (UniqueName: \"kubernetes.io/projected/9e4355c3-c0aa-4c59-9614-ad46da3f012a-kube-api-access-vsqxk\") pod \"multus-additional-cni-plugins-dgdjb\" (UID: \"9e4355c3-c0aa-4c59-9614-ad46da3f012a\") " pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.154676 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.154618 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrqk\" (UniqueName: \"kubernetes.io/projected/4c4f1949-5ea4-4839-9ccd-a3187ec13124-kube-api-access-6qrqk\") pod \"iptables-alerter-zzfph\" (UID: \"4c4f1949-5ea4-4839-9ccd-a3187ec13124\") " pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.154778 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.154755 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh92\" (UniqueName: \"kubernetes.io/projected/1f313f82-0e16-4e1c-8888-727cf6ea662c-kube-api-access-czh92\") pod \"node-ca-mbbgp\" (UID: \"1f313f82-0e16-4e1c-8888-727cf6ea662c\") " pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.235808 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.235745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" Jul 02 19:53:43.243927 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.243906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:53:43.253787 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.253767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:53:43.258792 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.258765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pb449" Jul 02 19:53:43.266170 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.266154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mbbgp" Jul 02 19:53:43.274947 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.274929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qw9rz" Jul 02 19:53:43.280727 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.280708 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gmft6" Jul 02 19:53:43.287038 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.287022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" Jul 02 19:53:43.293490 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.293474 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzfph" Jul 02 19:53:43.547109 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.547041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:43.547240 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.547157 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:43.547240 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.547207 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret podName:48751f64-539d-4c0f-a40f-84264b0b6720 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:44.547194048 +0000 UTC m=+3.915930751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret") pod "global-pull-secret-syncer-g7p4v" (UID: "48751f64-539d-4c0f-a40f-84264b0b6720") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:43.647688 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.647657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:43.647871 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.647733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:43.647871 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.647829 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:43.647871 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.647858 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:53:43.647871 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.647871 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:53:43.648078 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.647880 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7d874 for pod openshift-network-diagnostics/network-check-target-8ngzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:43.648078 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.647912 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs podName:d9291347-9dc5-48e1-b2af-ea6db41aa368 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:44.6478897 +0000 UTC m=+4.016626422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs") pod "network-metrics-daemon-gm4b8" (UID: "d9291347-9dc5-48e1-b2af-ea6db41aa368") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:43.648078 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:43.647933 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874 podName:dea8b301-09ab-4de3-93fe-f4cc9c01dbab nodeName:}" failed. No retries permitted until 2026-07-02 19:53:44.647922578 +0000 UTC m=+4.016659285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d874" (UniqueName: "kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874") pod "network-check-target-8ngzh" (UID: "dea8b301-09ab-4de3-93fe-f4cc9c01dbab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:43.880354 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:43.880324 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f313f82_0e16_4e1c_8888_727cf6ea662c.slice/crio-77923c4e14e53ed26ad54fd90cbec640039567bbd993d19645c88bdeaf05de64 WatchSource:0}: Error finding container 77923c4e14e53ed26ad54fd90cbec640039567bbd993d19645c88bdeaf05de64: Status 404 returned error can't find the container with id 77923c4e14e53ed26ad54fd90cbec640039567bbd993d19645c88bdeaf05de64 Jul 02 19:53:43.881892 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:43.881868 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75a85a2_4ff5_42ec_938b_e9f3e343d2ae.slice/crio-126d9cd1329ac3826655db7fa5e19f903d2003acc0c33dac32e7051c45401f47 WatchSource:0}: Error finding container 126d9cd1329ac3826655db7fa5e19f903d2003acc0c33dac32e7051c45401f47: Status 404 returned error can't find the container with id 126d9cd1329ac3826655db7fa5e19f903d2003acc0c33dac32e7051c45401f47 Jul 02 19:53:43.883124 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:43.883104 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4f1949_5ea4_4839_9ccd_a3187ec13124.slice/crio-183f3a44077e100506a3ab425d510f6b718d977300aaf2ded3c1a656261a00b4 WatchSource:0}: Error finding container 183f3a44077e100506a3ab425d510f6b718d977300aaf2ded3c1a656261a00b4: Status 404 returned error can't find the container with id 183f3a44077e100506a3ab425d510f6b718d977300aaf2ded3c1a656261a00b4 Jul 02 19:53:43.885587 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:43.885268 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdef8f6_2d6a_4491_ae2e_133524b09d1e.slice/crio-4fd5e59c3217d0a2c0acb33f96e95b14a1288d26fed5bfa713aece8f029d8e99 WatchSource:0}: Error finding container 4fd5e59c3217d0a2c0acb33f96e95b14a1288d26fed5bfa713aece8f029d8e99: Status 404 returned error can't find the container with id 4fd5e59c3217d0a2c0acb33f96e95b14a1288d26fed5bfa713aece8f029d8e99 Jul 02 19:53:43.886095 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:43.886077 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c54e50_2fb0_439f_89a2_19719be80f90.slice/crio-04e97364b625746bb481004719a641ca90a378c46b5214047f849b367f373450 WatchSource:0}: Error finding container 04e97364b625746bb481004719a641ca90a378c46b5214047f849b367f373450: Status 404 returned error can't find the container with id 04e97364b625746bb481004719a641ca90a378c46b5214047f849b367f373450 Jul 02 19:53:43.887698 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:43.887675 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe97c83_69a0_4435_a53b_2d32b8ba330b.slice/crio-38909413a2c403b59c706228f0a0b247ce02e05ea5a2f5c7c9805a79877fb7a0 WatchSource:0}: Error finding container 38909413a2c403b59c706228f0a0b247ce02e05ea5a2f5c7c9805a79877fb7a0: Status 404 returned error can't find the container with id 38909413a2c403b59c706228f0a0b247ce02e05ea5a2f5c7c9805a79877fb7a0 Jul 02 19:53:43.888450 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:53:43.888422 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4960089f_b7f6_4daa_ae08_c3d3df7cfe52.slice/crio-8663d07459844fe6d751aa398d13da05fd0b8eecf94a573191f398ac4307bb88 WatchSource:0}: Error finding container 8663d07459844fe6d751aa398d13da05fd0b8eecf94a573191f398ac4307bb88: Status 404 returned error can't find the container with id 8663d07459844fe6d751aa398d13da05fd0b8eecf94a573191f398ac4307bb88 Jul 02 19:53:43.968266 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.968216 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:48:42 +0000 UTC" deadline="2027-12-29 03:20:08.544212854 +0000 UTC" Jul 02 19:53:43.968589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:43.968267 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13063h26m24.575950452s" Jul 02 19:53:44.048816 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.048791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:44.048940 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.048897 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:44.057115 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.057086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gmft6" event={"ID":"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741","Type":"ContainerStarted","Data":"cb3289136f9f9ea88a961e064efa845b82a05ded0c0a809fbbe5c647aa7d60d8"} Jul 02 19:53:44.058182 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.058158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qw9rz" event={"ID":"4960089f-b7f6-4daa-ae08-c3d3df7cfe52","Type":"ContainerStarted","Data":"8663d07459844fe6d751aa398d13da05fd0b8eecf94a573191f398ac4307bb88"} Jul 02 19:53:44.059413 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.059384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"04e97364b625746bb481004719a641ca90a378c46b5214047f849b367f373450"} Jul 02 19:53:44.060394 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.060374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wdjp6" event={"ID":"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae","Type":"ContainerStarted","Data":"126d9cd1329ac3826655db7fa5e19f903d2003acc0c33dac32e7051c45401f47"} Jul 02 19:53:44.062258 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.062232 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" event={"ID":"c581f67582f2b7493459f83b250c1488","Type":"ContainerStarted","Data":"0c51e59cba586a38e38777726808dbef2b2118c0c6b5842253cd8279af96160a"} Jul 02 19:53:44.063385 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.063364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerStarted","Data":"5b16b3801e7a5139ef9f9867b1ddec6dd3a8132e46c70e6cf690c73ba9362019"} Jul 02 19:53:44.064615 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.064587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" event={"ID":"afe97c83-69a0-4435-a53b-2d32b8ba330b","Type":"ContainerStarted","Data":"38909413a2c403b59c706228f0a0b247ce02e05ea5a2f5c7c9805a79877fb7a0"} Jul 02 19:53:44.066105 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.066081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pb449" event={"ID":"ecdef8f6-2d6a-4491-ae2e-133524b09d1e","Type":"ContainerStarted","Data":"4fd5e59c3217d0a2c0acb33f96e95b14a1288d26fed5bfa713aece8f029d8e99"} Jul 02 19:53:44.067339 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.067319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzfph" event={"ID":"4c4f1949-5ea4-4839-9ccd-a3187ec13124","Type":"ContainerStarted","Data":"183f3a44077e100506a3ab425d510f6b718d977300aaf2ded3c1a656261a00b4"} Jul 02 19:53:44.068395 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.068376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mbbgp" event={"ID":"1f313f82-0e16-4e1c-8888-727cf6ea662c","Type":"ContainerStarted","Data":"77923c4e14e53ed26ad54fd90cbec640039567bbd993d19645c88bdeaf05de64"} Jul 02 19:53:44.077045 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.076995 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" podStartSLOduration=2.076984235 podStartE2EDuration="2.076984235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:53:44.076778048 +0000 UTC m=+3.445514772" watchObservedRunningTime="2026-07-02 19:53:44.076984235 +0000 UTC m=+3.445720960" Jul 02 19:53:44.559181 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.554242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:44.559181 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.554442 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:44.559181 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.554521 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret podName:48751f64-539d-4c0f-a40f-84264b0b6720 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:46.554502073 +0000 UTC m=+5.923238784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret") pod "global-pull-secret-syncer-g7p4v" (UID: "48751f64-539d-4c0f-a40f-84264b0b6720") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:44.655674 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.655637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:44.655854 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.655696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:44.655854 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.655848 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:44.655965 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.655908 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs podName:d9291347-9dc5-48e1-b2af-ea6db41aa368 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:46.655890577 +0000 UTC m=+6.024627293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs") pod "network-metrics-daemon-gm4b8" (UID: "d9291347-9dc5-48e1-b2af-ea6db41aa368") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:44.656335 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.656303 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:53:44.656335 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.656327 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:53:44.656502 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.656341 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7d874 for pod openshift-network-diagnostics/network-check-target-8ngzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:44.656502 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:44.656382 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874 podName:dea8b301-09ab-4de3-93fe-f4cc9c01dbab nodeName:}" failed. No retries permitted until 2026-07-02 19:53:46.656369265 +0000 UTC m=+6.025105971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d874" (UniqueName: "kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874") pod "network-check-target-8ngzh" (UID: "dea8b301-09ab-4de3-93fe-f4cc9c01dbab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:44.969392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.969248 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:48:42 +0000 UTC" deadline="2028-03-15 17:08:29.021816756 +0000 UTC" Jul 02 19:53:44.969392 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:44.969288 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14925h14m44.052533351s" Jul 02 19:53:45.049513 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:45.049453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:45.049677 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:45.049592 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:45.052276 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:45.052182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:45.052420 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:45.052293 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:45.094427 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:45.094389 2564 generic.go:356] "Generic (PLEG): container finished" podID="176a852dc8aa3e063105495696527f75" containerID="e73595b2b1afef2c35096fd7a904b1926e421fc819371505465634da450c82dd" exitCode=0 Jul 02 19:53:45.095695 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:45.095397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" event={"ID":"176a852dc8aa3e063105495696527f75","Type":"ContainerDied","Data":"e73595b2b1afef2c35096fd7a904b1926e421fc819371505465634da450c82dd"} Jul 02 19:53:46.049565 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:46.049534 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:46.050046 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.049670 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:46.117640 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:46.117603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" event={"ID":"176a852dc8aa3e063105495696527f75","Type":"ContainerStarted","Data":"b39fb2c80276e7edc393d93b23b48e7333e1f8648cc3e9be358ddb6d5e1c23f4"} Jul 02 19:53:46.572222 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:46.572175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:46.572436 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.572325 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:46.572436 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.572388 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret podName:48751f64-539d-4c0f-a40f-84264b0b6720 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:50.572371558 +0000 UTC m=+9.941108264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret") pod "global-pull-secret-syncer-g7p4v" (UID: "48751f64-539d-4c0f-a40f-84264b0b6720") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:46.673683 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:46.673494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:46.673683 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:46.673604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:46.674017 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.673752 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:53:46.674017 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.673773 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:53:46.674017 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.673786 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7d874 for pod openshift-network-diagnostics/network-check-target-8ngzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:46.674017 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.673844 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874 podName:dea8b301-09ab-4de3-93fe-f4cc9c01dbab nodeName:}" failed. No retries permitted until 2026-07-02 19:53:50.673826166 +0000 UTC m=+10.042562886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d874" (UniqueName: "kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874") pod "network-check-target-8ngzh" (UID: "dea8b301-09ab-4de3-93fe-f4cc9c01dbab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:46.674236 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.674226 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:46.674376 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:46.674285 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs podName:d9291347-9dc5-48e1-b2af-ea6db41aa368 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:50.67425715 +0000 UTC m=+10.042993871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs") pod "network-metrics-daemon-gm4b8" (UID: "d9291347-9dc5-48e1-b2af-ea6db41aa368") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:47.049648 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:47.049571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:47.050095 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:47.049700 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:47.050095 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:47.050090 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:47.050215 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:47.050180 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:48.049370 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:48.048873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:48.049370 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:48.049022 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:49.052697 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:49.049184 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:49.052697 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:49.049311 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:49.052697 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:49.049685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:49.052697 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:49.049775 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:50.048848 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:50.048809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:50.049047 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.048954 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:50.605958 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:50.605918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:50.606413 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.606119 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:50.606413 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.606200 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret podName:48751f64-539d-4c0f-a40f-84264b0b6720 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:58.606179842 +0000 UTC m=+17.974916559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret") pod "global-pull-secret-syncer-g7p4v" (UID: "48751f64-539d-4c0f-a40f-84264b0b6720") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:50.706590 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:50.706554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:50.706783 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:50.706614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:50.706783 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.706741 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:50.706783 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.706756 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:53:50.706783 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.706781 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:53:50.706965 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.706796 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7d874 for pod openshift-network-diagnostics/network-check-target-8ngzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:50.706965 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.706811 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs podName:d9291347-9dc5-48e1-b2af-ea6db41aa368 nodeName:}" failed. No retries permitted until 2026-07-02 19:53:58.706790932 +0000 UTC m=+18.075527654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs") pod "network-metrics-daemon-gm4b8" (UID: "d9291347-9dc5-48e1-b2af-ea6db41aa368") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:50.706965 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:50.706848 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874 podName:dea8b301-09ab-4de3-93fe-f4cc9c01dbab nodeName:}" failed. No retries permitted until 2026-07-02 19:53:58.706832045 +0000 UTC m=+18.075568764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d874" (UniqueName: "kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874") pod "network-check-target-8ngzh" (UID: "dea8b301-09ab-4de3-93fe-f4cc9c01dbab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:51.050238 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:51.049726 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:51.050238 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:51.049836 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:51.050238 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:51.049869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:51.050238 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:51.049978 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:52.048910 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:52.048849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:52.049407 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:52.049001 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:53.049254 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:53.049221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:53.049710 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:53.049356 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:53.049710 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:53.049429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:53.049710 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:53.049574 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:54.049469 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:54.049429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:54.049807 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:54.049585 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:55.049182 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:55.049154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:55.049345 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:55.049279 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:55.049421 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:55.049351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:55.049498 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:55.049455 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:56.049246 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:56.049210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:56.049413 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:56.049344 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:57.049007 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:57.048947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:57.049007 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:57.048998 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:57.049519 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:57.049087 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:53:57.052533 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:57.049591 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:58.048871 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:58.048841 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:58.049048 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.048952 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:53:58.666340 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:58.666286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:58.666530 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.666489 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:58.666593 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.666561 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret podName:48751f64-539d-4c0f-a40f-84264b0b6720 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:14.666545322 +0000 UTC m=+34.035282029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret") pod "global-pull-secret-syncer-g7p4v" (UID: "48751f64-539d-4c0f-a40f-84264b0b6720") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:53:58.766849 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:58.766807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:58.767036 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:58.766869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:53:58.767036 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.767006 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:58.767148 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.767071 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs podName:d9291347-9dc5-48e1-b2af-ea6db41aa368 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:14.767053422 +0000 UTC m=+34.135790145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs") pod "network-metrics-daemon-gm4b8" (UID: "d9291347-9dc5-48e1-b2af-ea6db41aa368") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:53:58.767148 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.767008 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:53:58.767148 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.767136 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:53:58.767341 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.767151 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7d874 for pod openshift-network-diagnostics/network-check-target-8ngzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:58.767341 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:58.767209 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874 podName:dea8b301-09ab-4de3-93fe-f4cc9c01dbab nodeName:}" failed. No retries permitted until 2026-07-02 19:54:14.76719391 +0000 UTC m=+34.135930615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d874" (UniqueName: "kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874") pod "network-check-target-8ngzh" (UID: "dea8b301-09ab-4de3-93fe-f4cc9c01dbab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:53:59.049146 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:59.049065 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:53:59.049586 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:59.049199 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:53:59.049586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:53:59.049252 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:53:59.049586 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:53:59.049362 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:00.048810 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:00.048774 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:00.048986 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:00.048919 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:01.052229 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:01.052188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:01.052707 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:01.052338 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:01.053403 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:01.052831 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:01.053403 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:01.053001 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:02.048962 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:02.048929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:02.049128 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:02.049073 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:03.049205 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.048913 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:03.049706 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.048914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:03.049706 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:03.049289 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:03.049706 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:03.049378 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:03.153423 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.153396 2564 generic.go:356] "Generic (PLEG): container finished" podID="9e4355c3-c0aa-4c59-9614-ad46da3f012a" containerID="676bfcd25592d72238925a154498d51a6cf24d3c1564ac0fe3c27c805f351e37" exitCode=0 Jul 02 19:54:03.153557 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.153493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerDied","Data":"676bfcd25592d72238925a154498d51a6cf24d3c1564ac0fe3c27c805f351e37"} Jul 02 19:54:03.155065 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.155033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" event={"ID":"afe97c83-69a0-4435-a53b-2d32b8ba330b","Type":"ContainerStarted","Data":"7e4f47f49abb010ec7a97a29d2d764af8d06916d4b497fa522a271dfb1c84ddd"} Jul 02 19:54:03.156482 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.156441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pb449" event={"ID":"ecdef8f6-2d6a-4491-ae2e-133524b09d1e","Type":"ContainerStarted","Data":"9e9303ee21d8a5625f507008cff2a62608ae30480d53972fef6a44d24fb089b3"} Jul 02 19:54:03.157801 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.157781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mbbgp" event={"ID":"1f313f82-0e16-4e1c-8888-727cf6ea662c","Type":"ContainerStarted","Data":"8e59a9f0eb0fe14ded14ab0df598d855af8c04e875973572879955422658d776"} Jul 02 19:54:03.159133 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.159115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gmft6" event={"ID":"24d25ab8-fc60-44d8-9a1b-e6eddd2f1741","Type":"ContainerStarted","Data":"5f1a9331a3c4b33323668688552d0c8f1b909fb0e4b87a29fefb3a59460fab29"} Jul 02 19:54:03.160518 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.160496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qw9rz" event={"ID":"4960089f-b7f6-4daa-ae08-c3d3df7cfe52","Type":"ContainerStarted","Data":"5ab6d010aaf1f86c729f927d26ab60f6917a71a70956f8dd62528d35008823e2"} Jul 02 19:54:03.165010 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.164992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"d35d78676b0fba095f912cba4b9ef31413d35aeb177f82302b464a8d237fa474"} Jul 02 19:54:03.165105 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.165050 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"bb15da08ad809bbd4624a58ce0349187bc81229696a794b8172f627ab23f0eee"} Jul 02 19:54:03.165105 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.165095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"12979ca3b638c45945259cadf0ac2e13e7557a70d550acbff940fa361b861e1f"} Jul 02 19:54:03.165211 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.165133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"d4db76d96474a3aff8c32a9bd1689fc099177b8e2a0f52fa3601ce8b42aeccde"} Jul 02 19:54:03.165211 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.165168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"e693feb01159d2b11f78eb05a992c07df99a82fd676eccb665a50b151510d208"} Jul 02 19:54:03.166232 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.166212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wdjp6" event={"ID":"f75a85a2-4ff5-42ec-938b-e9f3e343d2ae","Type":"ContainerStarted","Data":"20d2eff3953f53ecc7988665819f900af3c5296ba37be7835ec8c5327af946c7"} Jul 02 19:54:03.179287 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.179250 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" podStartSLOduration=21.179237858 podStartE2EDuration="21.179237858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:53:46.137006818 +0000 UTC m=+5.505743544" watchObservedRunningTime="2026-07-02 19:54:03.179237858 +0000 UTC m=+22.547974582" Jul 02 19:54:03.197149 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.197116 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pb449" podStartSLOduration=3.5054115489999997 podStartE2EDuration="22.197105669s" totalImagesPullingTime="18.69169412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:03.196986182 +0000 UTC m=+22.565722907" watchObservedRunningTime="2026-07-02 19:54:03.197105669 +0000 UTC m=+22.565842392" Jul 02 19:54:03.211373 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.211304 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mbbgp" podStartSLOduration=3.522490714 podStartE2EDuration="22.211288794s" totalImagesPullingTime="18.68879808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:03.210860434 +0000 UTC m=+22.579597159" watchObservedRunningTime="2026-07-02 19:54:03.211288794 +0000 UTC m=+22.580025582" Jul 02 19:54:03.230246 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.230208 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gmft6" podStartSLOduration=3.47975551 podStartE2EDuration="22.230196273s" totalImagesPullingTime="18.750440763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:03.229601391 +0000 UTC m=+22.598338116" watchObservedRunningTime="2026-07-02 19:54:03.230196273 +0000 UTC m=+22.598932997" Jul 02 19:54:03.248149 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:03.248113 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wdjp6" podStartSLOduration=8.213455012 podStartE2EDuration="22.248099702s" totalImagesPullingTime="14.03464469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:03.248095936 +0000 UTC m=+22.616832660" watchObservedRunningTime="2026-07-02 19:54:03.248099702 +0000 UTC m=+22.616836427" Jul 02 19:54:04.048580 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:04.048547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:04.048841 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:04.048708 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:04.170158 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:04.170124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzfph" event={"ID":"4c4f1949-5ea4-4839-9ccd-a3187ec13124","Type":"ContainerStarted","Data":"632ba41f6399fdf8d9b03e0a15bc40fd7a06dc8d17dea1cc17380cb9c36b53b0"} Jul 02 19:54:04.174709 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:04.174676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"ee7b1900d2bf07a07a00376122758f113e8a4474af145353406f483f2df46bc2"} Jul 02 19:54:04.185809 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:04.185757 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qw9rz" podStartSLOduration=4.499833405 podStartE2EDuration="23.185738434s" totalImagesPullingTime="18.685905029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:03.265203748 +0000 UTC m=+22.633940470" watchObservedRunningTime="2026-07-02 19:54:04.185738434 +0000 UTC m=+23.554475161" Jul 02 19:54:04.186651 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:04.186612 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zzfph" podStartSLOduration=4.497813193 podStartE2EDuration="23.18659614s" totalImagesPullingTime="18.688782947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:04.185980043 +0000 UTC m=+23.554716770" watchObservedRunningTime="2026-07-02 19:54:04.18659614 +0000 UTC m=+23.555332868" Jul 02 19:54:04.262237 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:04.262024 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 19:54:04.998834 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:04.998687 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T19:54:04.262224515Z","UUID":"abd9c1b2-a1fa-401c-9ca0-b566636b766b","Handler":null,"Name":"","Endpoint":""} Jul 02 19:54:05.001844 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:05.001819 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 19:54:05.002003 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:05.001865 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 19:54:05.049352 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:05.049284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:05.049609 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:05.049417 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:05.049609 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:05.049552 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:05.049760 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:05.049682 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:05.179546 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:05.179510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" event={"ID":"afe97c83-69a0-4435-a53b-2d32b8ba330b","Type":"ContainerStarted","Data":"018c7f64aa74b4f1fb02316b3d6d6b76d831804a58830823a8d260d76c4cfd1d"} Jul 02 19:54:06.048566 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:06.048529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:06.048758 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:06.048655 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:06.185706 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:06.185666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"a80f20df5f1a942179d1e47e1e8fb1ba005da5ecf5bfd6af2a6460a14660ec41"} Jul 02 19:54:06.188041 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:06.188012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" event={"ID":"afe97c83-69a0-4435-a53b-2d32b8ba330b","Type":"ContainerStarted","Data":"9e945eab3bf5fffe4c5df409382d00ac971b68fd0540be3ba568930d96517f9e"} Jul 02 19:54:06.205817 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:06.205776 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pl2xc" podStartSLOduration=3.85517662 podStartE2EDuration="25.205763075s" totalImagesPullingTime="21.350586455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:06.205723474 +0000 UTC m=+25.574460199" watchObservedRunningTime="2026-07-02 19:54:06.205763075 +0000 UTC m=+25.574499800" Jul 02 19:54:06.586043 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:06.586007 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:54:06.586970 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:06.586952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:54:07.049016 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:07.048984 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:07.049213 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:07.048983 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:07.049213 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:07.049128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:07.049332 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:07.049215 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:08.048663 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.048638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:08.049141 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:08.048749 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:08.197545 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.197230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" event={"ID":"53c54e50-2fb0-439f-89a2-19719be80f90","Type":"ContainerStarted","Data":"48252b0e137596b434cad34870257838b0b6df91a3e936b0aa11a1e18b131716"} Jul 02 19:54:08.197856 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.197565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:54:08.197856 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.197618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:54:08.197856 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.197664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:54:08.215523 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.215325 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:54:08.215523 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.215420 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:54:08.226319 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:08.226279 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" podStartSLOduration=8.47999355 podStartE2EDuration="27.226265707s" totalImagesPullingTime="18.746272157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:08.224781711 +0000 UTC m=+27.593518447" watchObservedRunningTime="2026-07-02 19:54:08.226265707 +0000 UTC m=+27.595002431" Jul 02 19:54:09.049415 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:09.049379 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:09.049834 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:09.049394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:09.049834 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:09.049518 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:09.049834 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:09.049628 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:09.201326 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:09.201292 2564 generic.go:356] "Generic (PLEG): container finished" podID="9e4355c3-c0aa-4c59-9614-ad46da3f012a" containerID="4492467f095a36af2c95b55f6826e5792357fe7467a2a439a2a9cf56b8a40664" exitCode=0 Jul 02 19:54:09.201528 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:09.201343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerDied","Data":"4492467f095a36af2c95b55f6826e5792357fe7467a2a439a2a9cf56b8a40664"} Jul 02 19:54:10.049327 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.049149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:10.049531 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:10.049407 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:10.100396 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.100362 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7p4v"] Jul 02 19:54:10.100568 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.100455 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:10.100568 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:10.100550 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:10.103352 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.103326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gm4b8"] Jul 02 19:54:10.103938 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.103908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8ngzh"] Jul 02 19:54:10.104046 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.103996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:10.104120 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:10.104098 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:10.203507 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.203482 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:10.203638 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:10.203589 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:10.448258 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.448221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:54:10.448451 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.448379 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 02 19:54:10.448855 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:10.448833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wdjp6" Jul 02 19:54:11.207087 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:11.207057 2564 generic.go:356] "Generic (PLEG): container finished" podID="9e4355c3-c0aa-4c59-9614-ad46da3f012a" containerID="2a0b84723eaf62272f1e53294a508ceff7d358f6b3e39dc6e3f6b350fe1b0b85" exitCode=0 Jul 02 19:54:11.207632 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:11.207145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerDied","Data":"2a0b84723eaf62272f1e53294a508ceff7d358f6b3e39dc6e3f6b350fe1b0b85"} Jul 02 19:54:12.049137 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:12.048900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:12.049137 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:12.049034 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:12.049137 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:12.049118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:12.049404 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:12.049195 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:12.049404 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:12.049248 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:12.049404 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:12.049336 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:12.212628 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:12.212597 2564 generic.go:356] "Generic (PLEG): container finished" podID="9e4355c3-c0aa-4c59-9614-ad46da3f012a" containerID="c091cad25cb05cf060c3245bc9c62206c3191503b79e4c74aea5119fcaa6c0e7" exitCode=0 Jul 02 19:54:12.213238 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:12.212668 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerDied","Data":"c091cad25cb05cf060c3245bc9c62206c3191503b79e4c74aea5119fcaa6c0e7"} Jul 02 19:54:14.048967 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:14.048934 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:14.048967 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:14.048954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:14.048967 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:14.048941 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:14.049532 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.049056 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7p4v" podUID="48751f64-539d-4c0f-a40f-84264b0b6720" Jul 02 19:54:14.049532 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.049148 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gm4b8" podUID="d9291347-9dc5-48e1-b2af-ea6db41aa368" Jul 02 19:54:14.049532 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.049182 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8ngzh" podUID="dea8b301-09ab-4de3-93fe-f4cc9c01dbab" Jul 02 19:54:14.687040 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:14.686850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:14.687233 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.687010 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:54:14.687233 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.687123 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret podName:48751f64-539d-4c0f-a40f-84264b0b6720 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:46.687108331 +0000 UTC m=+66.055845034 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret") pod "global-pull-secret-syncer-g7p4v" (UID: "48751f64-539d-4c0f-a40f-84264b0b6720") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:54:14.787886 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:14.787843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:14.788068 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:14.787899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:14.788068 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.787994 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:54:14.788068 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.787997 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:54:14.788068 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.788024 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:54:14.788068 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.788036 2564 projected.go:196] Error preparing data for projected volume kube-api-access-7d874 for pod openshift-network-diagnostics/network-check-target-8ngzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:54:14.788068 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.788057 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs podName:d9291347-9dc5-48e1-b2af-ea6db41aa368 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:46.788043018 +0000 UTC m=+66.156779721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs") pod "network-metrics-daemon-gm4b8" (UID: "d9291347-9dc5-48e1-b2af-ea6db41aa368") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:54:14.788345 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:14.788084 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874 podName:dea8b301-09ab-4de3-93fe-f4cc9c01dbab nodeName:}" failed. No retries permitted until 2026-07-02 19:54:46.788068198 +0000 UTC m=+66.156804915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d874" (UniqueName: "kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874") pod "network-check-target-8ngzh" (UID: "dea8b301-09ab-4de3-93fe-f4cc9c01dbab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:54:15.467832 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.467803 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeReady" Jul 02 19:54:15.468254 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.467987 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 19:54:15.523543 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.523443 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5984fbd7f-9tc9p"] Jul 02 19:54:15.529226 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.529194 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w6qq7"] Jul 02 19:54:15.529526 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.529502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.532824 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.532789 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 19:54:15.532824 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.532811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 19:54:15.532977 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.532848 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 19:54:15.533149 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.533131 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kcm5k"] Jul 02 19:54:15.533267 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.533246 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hfqbh\"" Jul 02 19:54:15.533340 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.533258 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.536995 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.536978 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:15.537102 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.537002 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nzdvh\"" Jul 02 19:54:15.537102 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.537088 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 19:54:15.537294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.537278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 19:54:15.540016 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.539828 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 19:54:15.540016 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.539852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 19:54:15.540016 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.539829 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 19:54:15.540474 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.540445 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tjpqs\"" Jul 02 19:54:15.540729 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.540712 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 19:54:15.544716 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.544695 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5984fbd7f-9tc9p"] Jul 02 19:54:15.548316 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.547563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcm5k"] Jul 02 19:54:15.550399 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.550377 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6qq7"] Jul 02 19:54:15.694331 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-certificates\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.694331 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-bound-sa-token\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.694604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856c4e6a-55d7-414e-b534-a6d36dc652f3-tmp-dir\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.694604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27g4f\" (UniqueName: \"kubernetes.io/projected/a1d7c432-ae9a-43e2-b4cb-93dca17f6a67-kube-api-access-27g4f\") pod \"ingress-canary-kcm5k\" (UID: \"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67\") " pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:15.694604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-trusted-ca\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.694604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-image-registry-private-configuration\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.694604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwk84\" (UniqueName: \"kubernetes.io/projected/856c4e6a-55d7-414e-b534-a6d36dc652f3-kube-api-access-nwk84\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.694604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1d7c432-ae9a-43e2-b4cb-93dca17f6a67-cert\") pod \"ingress-canary-kcm5k\" (UID: \"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67\") " pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:15.694604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de5b2d34-bd4c-4036-a4be-9b1f928a0346-ca-trust-extracted\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.694887 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-installation-pull-secrets\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.694887 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/856c4e6a-55d7-414e-b534-a6d36dc652f3-config-volume\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.694887 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.694887 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694789 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.694887 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.694840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhs2q\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-kube-api-access-rhs2q\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.795688 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-image-registry-private-configuration\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.795688 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwk84\" (UniqueName: \"kubernetes.io/projected/856c4e6a-55d7-414e-b534-a6d36dc652f3-kube-api-access-nwk84\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.795688 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1d7c432-ae9a-43e2-b4cb-93dca17f6a67-cert\") pod \"ingress-canary-kcm5k\" (UID: \"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67\") " pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:15.795957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de5b2d34-bd4c-4036-a4be-9b1f928a0346-ca-trust-extracted\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.795957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-installation-pull-secrets\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.795957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/856c4e6a-55d7-414e-b534-a6d36dc652f3-config-volume\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.795957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.795957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.795803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.795957 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:15.795906 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:54:15.795957 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:15.795920 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5984fbd7f-9tc9p: secret "image-registry-tls" not found Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:15.795986 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls podName:de5b2d34-bd4c-4036-a4be-9b1f928a0346 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:16.295964977 +0000 UTC m=+35.664701700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls") pod "image-registry-5984fbd7f-9tc9p" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346") : secret "image-registry-tls" not found Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:15.795992 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:15.796061 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls podName:856c4e6a-55d7-414e-b534-a6d36dc652f3 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:16.296041562 +0000 UTC m=+35.664778270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls") pod "dns-default-w6qq7" (UID: "856c4e6a-55d7-414e-b534-a6d36dc652f3") : secret "dns-default-metrics-tls" not found Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhs2q\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-kube-api-access-rhs2q\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-certificates\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-bound-sa-token\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856c4e6a-55d7-414e-b534-a6d36dc652f3-tmp-dir\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796237 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de5b2d34-bd4c-4036-a4be-9b1f928a0346-ca-trust-extracted\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.796294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27g4f\" (UniqueName: \"kubernetes.io/projected/a1d7c432-ae9a-43e2-b4cb-93dca17f6a67-kube-api-access-27g4f\") pod \"ingress-canary-kcm5k\" (UID: \"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67\") " pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:15.796752 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-trusted-ca\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.796752 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796328 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/856c4e6a-55d7-414e-b534-a6d36dc652f3-config-volume\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.796752 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.796606 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856c4e6a-55d7-414e-b534-a6d36dc652f3-tmp-dir\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.797360 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.797340 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-certificates\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.797624 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.797606 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-trusted-ca\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.800455 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.800429 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-image-registry-private-configuration\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.800625 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.800529 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-installation-pull-secrets\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.800625 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.800537 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1d7c432-ae9a-43e2-b4cb-93dca17f6a67-cert\") pod \"ingress-canary-kcm5k\" (UID: \"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67\") " pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:15.804453 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.804431 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-bound-sa-token\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.804621 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.804600 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwk84\" (UniqueName: \"kubernetes.io/projected/856c4e6a-55d7-414e-b534-a6d36dc652f3-kube-api-access-nwk84\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:15.805309 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.805285 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhs2q\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-kube-api-access-rhs2q\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:15.805481 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.805438 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27g4f\" (UniqueName: \"kubernetes.io/projected/a1d7c432-ae9a-43e2-b4cb-93dca17f6a67-kube-api-access-27g4f\") pod \"ingress-canary-kcm5k\" (UID: \"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67\") " pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:15.856379 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:15.856345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kcm5k" Jul 02 19:54:16.049413 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.049328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:16.049413 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.049347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:16.049413 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.049350 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:16.052394 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.052370 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:54:16.052540 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.052375 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:54:16.052540 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.052531 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:54:16.053740 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.053718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:54:16.053847 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.053722 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k665n\"" Jul 02 19:54:16.053847 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.053724 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdnmq\"" Jul 02 19:54:16.238409 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.238381 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kcm5k"] Jul 02 19:54:16.243947 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:16.243908 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d7c432_ae9a_43e2_b4cb_93dca17f6a67.slice/crio-c0d292077947c6141c357331e9bcdaa283d371c5d94c2b11234778934340ce39 WatchSource:0}: Error finding container c0d292077947c6141c357331e9bcdaa283d371c5d94c2b11234778934340ce39: Status 404 returned error can't find the container with id c0d292077947c6141c357331e9bcdaa283d371c5d94c2b11234778934340ce39 Jul 02 19:54:16.301235 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.301006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:16.301235 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:16.301221 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:54:16.301368 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:16.301221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:16.301368 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:16.301304 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls podName:856c4e6a-55d7-414e-b534-a6d36dc652f3 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:17.301282734 +0000 UTC m=+36.670019463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls") pod "dns-default-w6qq7" (UID: "856c4e6a-55d7-414e-b534-a6d36dc652f3") : secret "dns-default-metrics-tls" not found Jul 02 19:54:16.301446 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:16.301305 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:54:16.301446 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:16.301393 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5984fbd7f-9tc9p: secret "image-registry-tls" not found Jul 02 19:54:16.301528 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:16.301449 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls podName:de5b2d34-bd4c-4036-a4be-9b1f928a0346 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:17.301430896 +0000 UTC m=+36.670167600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls") pod "image-registry-5984fbd7f-9tc9p" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346") : secret "image-registry-tls" not found Jul 02 19:54:17.229098 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:17.229063 2564 generic.go:356] "Generic (PLEG): container finished" podID="9e4355c3-c0aa-4c59-9614-ad46da3f012a" containerID="1e8c0904410fa8e4aa7f936e7029b1d1a788d5723143925baf58384b8463fdbe" exitCode=0 Jul 02 19:54:17.229530 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:17.229144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerDied","Data":"1e8c0904410fa8e4aa7f936e7029b1d1a788d5723143925baf58384b8463fdbe"} Jul 02 19:54:17.230742 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:17.230715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcm5k" event={"ID":"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67","Type":"ContainerStarted","Data":"c0d292077947c6141c357331e9bcdaa283d371c5d94c2b11234778934340ce39"} Jul 02 19:54:17.309132 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:17.309109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:17.309247 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:17.309156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:17.309308 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:17.309284 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:54:17.309308 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:17.309299 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5984fbd7f-9tc9p: secret "image-registry-tls" not found Jul 02 19:54:17.309418 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:17.309353 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls podName:de5b2d34-bd4c-4036-a4be-9b1f928a0346 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:19.309334429 +0000 UTC m=+38.678071149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls") pod "image-registry-5984fbd7f-9tc9p" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346") : secret "image-registry-tls" not found Jul 02 19:54:17.309418 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:17.309397 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:54:17.309553 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:17.309453 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls podName:856c4e6a-55d7-414e-b534-a6d36dc652f3 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:19.30944034 +0000 UTC m=+38.678177068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls") pod "dns-default-w6qq7" (UID: "856c4e6a-55d7-414e-b534-a6d36dc652f3") : secret "dns-default-metrics-tls" not found Jul 02 19:54:18.236574 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:18.236533 2564 generic.go:356] "Generic (PLEG): container finished" podID="9e4355c3-c0aa-4c59-9614-ad46da3f012a" containerID="53d2ca4535f191a249aac91771429fd09880b24cabb66b75c4dd8843c89617f0" exitCode=0 Jul 02 19:54:18.237020 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:18.236582 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerDied","Data":"53d2ca4535f191a249aac91771429fd09880b24cabb66b75c4dd8843c89617f0"} Jul 02 19:54:19.241679 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:19.241643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" event={"ID":"9e4355c3-c0aa-4c59-9614-ad46da3f012a","Type":"ContainerStarted","Data":"c5c7be2131c4f2c168c7657023d2590fd05dee150546a6bdbf5b564c41ee7fa8"} Jul 02 19:54:19.243094 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:19.243071 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kcm5k" event={"ID":"a1d7c432-ae9a-43e2-b4cb-93dca17f6a67","Type":"ContainerStarted","Data":"3fded36b9b808dd39f7b0ffaa02b00cd53ea278261538b9e6c197ce7e93cc896"} Jul 02 19:54:19.264525 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:19.264480 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dgdjb" podStartSLOduration=9.044814567 podStartE2EDuration="38.264451753s" totalImagesPullingTime="29.219637186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:19.262846022 +0000 UTC m=+38.631582746" watchObservedRunningTime="2026-07-02 19:54:19.264451753 +0000 UTC m=+38.633188477" Jul 02 19:54:19.280816 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:19.280772 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kcm5k" podStartSLOduration=2.055624387 podStartE2EDuration="4.280755026s" totalImagesPullingTime="2.225130639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:19.280347821 +0000 UTC m=+38.649084794" watchObservedRunningTime="2026-07-02 19:54:19.280755026 +0000 UTC m=+38.649491752" Jul 02 19:54:19.327818 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:19.327783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:19.327933 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:19.327846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:19.327996 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:19.327976 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:54:19.328047 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:19.327998 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5984fbd7f-9tc9p: secret "image-registry-tls" not found Jul 02 19:54:19.328115 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:19.328054 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls podName:de5b2d34-bd4c-4036-a4be-9b1f928a0346 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:23.328039254 +0000 UTC m=+42.696775972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls") pod "image-registry-5984fbd7f-9tc9p" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346") : secret "image-registry-tls" not found Jul 02 19:54:19.328177 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:19.328110 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:54:19.328229 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:19.328217 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls podName:856c4e6a-55d7-414e-b534-a6d36dc652f3 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:23.328195748 +0000 UTC m=+42.696932468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls") pod "dns-default-w6qq7" (UID: "856c4e6a-55d7-414e-b534-a6d36dc652f3") : secret "dns-default-metrics-tls" not found Jul 02 19:54:21.754965 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.754933 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hzwm5"] Jul 02 19:54:21.792363 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.792335 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzwm5"] Jul 02 19:54:21.792516 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.792486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:21.795279 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.795245 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 19:54:21.795399 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.795285 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 19:54:21.795399 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.795311 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 19:54:21.795399 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.795331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hs5xt\"" Jul 02 19:54:21.796355 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.796342 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 19:54:21.948674 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.948636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:21.948674 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.948676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d595560-99c7-447b-9685-9e432ee1fde4-crio-socket\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:21.948867 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.948710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d595560-99c7-447b-9685-9e432ee1fde4-data-volume\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:21.948867 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.948809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpgp\" (UniqueName: \"kubernetes.io/projected/1d595560-99c7-447b-9685-9e432ee1fde4-kube-api-access-tkpgp\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:21.948867 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:21.948855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.049643 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.049565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.049643 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.049631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.049846 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.049656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d595560-99c7-447b-9685-9e432ee1fde4-crio-socket\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.049846 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.049674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d595560-99c7-447b-9685-9e432ee1fde4-data-volume\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.049846 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.049713 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:54:22.049846 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.049772 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:54:22.049846 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.049772 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d595560-99c7-447b-9685-9e432ee1fde4-crio-socket\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.049846 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.049779 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:22.549761369 +0000 UTC m=+41.918498072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : secret "insights-runtime-extractor-tls" not found Jul 02 19:54:22.050151 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.049900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpgp\" (UniqueName: \"kubernetes.io/projected/1d595560-99c7-447b-9685-9e432ee1fde4-kube-api-access-tkpgp\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.050151 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.049976 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:22.549954401 +0000 UTC m=+41.918691113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : configmap "kube-rbac-proxy" not found Jul 02 19:54:22.050151 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.049986 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d595560-99c7-447b-9685-9e432ee1fde4-data-volume\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.059119 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.059096 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpgp\" (UniqueName: \"kubernetes.io/projected/1d595560-99c7-447b-9685-9e432ee1fde4-kube-api-access-tkpgp\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.553823 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.553785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.553994 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:22.553858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:22.553994 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.553933 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:54:22.553994 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.553947 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:54:22.554161 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.554000 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:23.55398433 +0000 UTC m=+42.922721034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : secret "insights-runtime-extractor-tls" not found Jul 02 19:54:22.554161 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:22.554015 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:23.554009313 +0000 UTC m=+42.922746015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : configmap "kube-rbac-proxy" not found Jul 02 19:54:23.359902 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:23.359863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:23.359902 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:23.359902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:23.360380 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.360037 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:54:23.360380 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.360047 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5984fbd7f-9tc9p: secret "image-registry-tls" not found Jul 02 19:54:23.360380 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.360044 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:54:23.360380 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.360094 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls podName:de5b2d34-bd4c-4036-a4be-9b1f928a0346 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:31.360080767 +0000 UTC m=+50.728817469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls") pod "image-registry-5984fbd7f-9tc9p" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346") : secret "image-registry-tls" not found Jul 02 19:54:23.360380 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.360107 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls podName:856c4e6a-55d7-414e-b534-a6d36dc652f3 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:31.360100633 +0000 UTC m=+50.728837336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls") pod "dns-default-w6qq7" (UID: "856c4e6a-55d7-414e-b534-a6d36dc652f3") : secret "dns-default-metrics-tls" not found Jul 02 19:54:23.545402 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:23.545372 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qw9rz_4960089f-b7f6-4daa-ae08-c3d3df7cfe52/dns-node-resolver/0.log" Jul 02 19:54:23.561688 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:23.561665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:23.561800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:23.561784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:23.561849 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.561789 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:54:23.561881 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.561858 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:54:23.561910 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.561897 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:25.561880139 +0000 UTC m=+44.930616856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : configmap "kube-rbac-proxy" not found Jul 02 19:54:23.561952 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:23.561913 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:25.561906288 +0000 UTC m=+44.930642991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : secret "insights-runtime-extractor-tls" not found Jul 02 19:54:24.226883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.226844 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-d5mhr"] Jul 02 19:54:24.246116 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.246090 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-d5mhr"] Jul 02 19:54:24.246265 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.246203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.248868 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.248844 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 02 19:54:24.248976 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.248951 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 02 19:54:24.250378 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.250356 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 02 19:54:24.250504 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.250390 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 02 19:54:24.250504 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.250361 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c9gqw\"" Jul 02 19:54:24.369800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.369767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdvf\" (UniqueName: \"kubernetes.io/projected/2acc10da-c5d3-40a0-996c-e11c77e5126c-kube-api-access-svdvf\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.369800 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.369799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2acc10da-c5d3-40a0-996c-e11c77e5126c-signing-key\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.370253 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.369837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2acc10da-c5d3-40a0-996c-e11c77e5126c-signing-cabundle\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.470660 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.470618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdvf\" (UniqueName: \"kubernetes.io/projected/2acc10da-c5d3-40a0-996c-e11c77e5126c-kube-api-access-svdvf\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.470660 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.470662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2acc10da-c5d3-40a0-996c-e11c77e5126c-signing-key\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.470908 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.470707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2acc10da-c5d3-40a0-996c-e11c77e5126c-signing-cabundle\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.471413 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.471387 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2acc10da-c5d3-40a0-996c-e11c77e5126c-signing-cabundle\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.473040 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.473020 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2acc10da-c5d3-40a0-996c-e11c77e5126c-signing-key\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.479077 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.479022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdvf\" (UniqueName: \"kubernetes.io/projected/2acc10da-c5d3-40a0-996c-e11c77e5126c-kube-api-access-svdvf\") pod \"service-ca-758f599c5-d5mhr\" (UID: \"2acc10da-c5d3-40a0-996c-e11c77e5126c\") " pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.544974 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.544952 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mbbgp_1f313f82-0e16-4e1c-8888-727cf6ea662c/node-ca/0.log" Jul 02 19:54:24.555209 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.555189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-d5mhr" Jul 02 19:54:24.698368 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.698339 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-d5mhr"] Jul 02 19:54:24.701399 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:24.701374 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acc10da_c5d3_40a0_996c_e11c77e5126c.slice/crio-32447a7d27590aeefc612edc4dcc497a2f6c370420acd4b5fc71f49da1e1014f WatchSource:0}: Error finding container 32447a7d27590aeefc612edc4dcc497a2f6c370420acd4b5fc71f49da1e1014f: Status 404 returned error can't find the container with id 32447a7d27590aeefc612edc4dcc497a2f6c370420acd4b5fc71f49da1e1014f Jul 02 19:54:24.946400 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:24.946371 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcm5k_a1d7c432-ae9a-43e2-b4cb-93dca17f6a67/serve-healthcheck-canary/0.log" Jul 02 19:54:25.258280 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:25.258194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-d5mhr" event={"ID":"2acc10da-c5d3-40a0-996c-e11c77e5126c","Type":"ContainerStarted","Data":"32447a7d27590aeefc612edc4dcc497a2f6c370420acd4b5fc71f49da1e1014f"} Jul 02 19:54:25.580610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:25.580522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:25.581076 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:25.580612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:25.581076 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:25.580703 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:54:25.581076 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:25.580789 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:29.580769063 +0000 UTC m=+48.949505782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : secret "insights-runtime-extractor-tls" not found Jul 02 19:54:25.581076 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:25.580710 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:54:25.581076 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:25.580857 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:29.580840573 +0000 UTC m=+48.949577280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : configmap "kube-rbac-proxy" not found Jul 02 19:54:27.264436 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:27.264395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-d5mhr" event={"ID":"2acc10da-c5d3-40a0-996c-e11c77e5126c","Type":"ContainerStarted","Data":"c0083e746348c139bd5482624380752dd699c48f408ebe8cb43093430a2f1b40"} Jul 02 19:54:27.323300 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:27.323249 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-d5mhr" podStartSLOduration=0.930873252 podStartE2EDuration="3.323231732s" totalImagesPullingTime="2.39235848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:27.32156776 +0000 UTC m=+46.690304501" watchObservedRunningTime="2026-07-02 19:54:27.323231732 +0000 UTC m=+46.691968453" Jul 02 19:54:29.614562 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:29.614525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:29.614997 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:29.614595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:29.614997 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:29.614689 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:54:29.614997 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:29.614763 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls podName:1d595560-99c7-447b-9685-9e432ee1fde4 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:37.614745739 +0000 UTC m=+56.983482442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzwm5" (UID: "1d595560-99c7-447b-9685-9e432ee1fde4") : secret "insights-runtime-extractor-tls" not found Jul 02 19:54:29.615124 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:29.615073 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d595560-99c7-447b-9685-9e432ee1fde4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:31.424892 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:31.424856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:31.425330 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:31.424907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") pod \"image-registry-5984fbd7f-9tc9p\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:31.425330 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:31.425015 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:54:31.425330 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:31.425081 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls podName:856c4e6a-55d7-414e-b534-a6d36dc652f3 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:47.425066014 +0000 UTC m=+66.793802730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls") pod "dns-default-w6qq7" (UID: "856c4e6a-55d7-414e-b534-a6d36dc652f3") : secret "dns-default-metrics-tls" not found Jul 02 19:54:31.425330 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:31.425111 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:54:31.425330 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:31.425134 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5984fbd7f-9tc9p: secret "image-registry-tls" not found Jul 02 19:54:31.425330 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:31.425194 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls podName:de5b2d34-bd4c-4036-a4be-9b1f928a0346 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:47.425176964 +0000 UTC m=+66.793913686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls") pod "image-registry-5984fbd7f-9tc9p" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346") : secret "image-registry-tls" not found Jul 02 19:54:37.677257 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:37.677209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:37.679560 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:37.679535 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d595560-99c7-447b-9685-9e432ee1fde4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzwm5\" (UID: \"1d595560-99c7-447b-9685-9e432ee1fde4\") " pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:37.702581 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:37.702551 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzwm5" Jul 02 19:54:37.842592 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:37.842562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzwm5"] Jul 02 19:54:37.845977 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:37.845937 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d595560_99c7_447b_9685_9e432ee1fde4.slice/crio-dee488d19c4231b9b1e4175ce53f26d188ccc901e476202219c51152d72dca2f WatchSource:0}: Error finding container dee488d19c4231b9b1e4175ce53f26d188ccc901e476202219c51152d72dca2f: Status 404 returned error can't find the container with id dee488d19c4231b9b1e4175ce53f26d188ccc901e476202219c51152d72dca2f Jul 02 19:54:38.294779 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:38.294655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzwm5" event={"ID":"1d595560-99c7-447b-9685-9e432ee1fde4","Type":"ContainerStarted","Data":"dee488d19c4231b9b1e4175ce53f26d188ccc901e476202219c51152d72dca2f"} Jul 02 19:54:40.218828 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:40.218800 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xj2jm" Jul 02 19:54:42.305605 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:42.305562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzwm5" event={"ID":"1d595560-99c7-447b-9685-9e432ee1fde4","Type":"ContainerStarted","Data":"25af7cefab4aa053fd3c90940921edd5110da725ba64010b87ed8dcfd32e49a1"} Jul 02 19:54:43.310476 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:43.310435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzwm5" event={"ID":"1d595560-99c7-447b-9685-9e432ee1fde4","Type":"ContainerStarted","Data":"d43356e6e46d6f7dfe8cc2015fe36643761bdb647fc246a6f6679fc4261b626b"} Jul 02 19:54:45.028440 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.028409 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb"] Jul 02 19:54:45.031490 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.031449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.031812 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.031782 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5"] Jul 02 19:54:45.034339 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.034314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 19:54:45.034339 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.034331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.034687 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.034669 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 19:54:45.035770 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.035751 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 02 19:54:45.035860 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.035753 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 19:54:45.036716 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.036688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 02 19:54:45.037135 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.037122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xbl87\"" Jul 02 19:54:45.043117 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.043084 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb"] Jul 02 19:54:45.044074 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.044050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5"] Jul 02 19:54:45.098552 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.098523 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5984fbd7f-9tc9p"] Jul 02 19:54:45.098758 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:45.098733 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" podUID="de5b2d34-bd4c-4036-a4be-9b1f928a0346" Jul 02 19:54:45.137830 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.137799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfcd50fd-8aa1-4d83-a472-d949312be4a3-tmp\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.137996 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.137838 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6947cb4444-6vzp5\" (UID: \"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.137996 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.137862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjct\" (UniqueName: \"kubernetes.io/projected/dfcd50fd-8aa1-4d83-a472-d949312be4a3-kube-api-access-lhjct\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.138116 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.138010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7dh\" (UniqueName: \"kubernetes.io/projected/3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9-kube-api-access-xz7dh\") pod \"managed-serviceaccount-addon-agent-6947cb4444-6vzp5\" (UID: \"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.138116 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.138065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfcd50fd-8aa1-4d83-a472-d949312be4a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.147268 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.147236 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f6c7f4bf-pvrmp"] Jul 02 19:54:45.151437 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.151415 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.164572 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.164546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f6c7f4bf-pvrmp"] Jul 02 19:54:45.239509 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed273e4-d733-4174-9881-e939528c79f2-image-registry-private-configuration\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.239509 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed273e4-d733-4174-9881-e939528c79f2-registry-certificates\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.239509 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-bound-sa-token\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.239773 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7dh\" (UniqueName: \"kubernetes.io/projected/3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9-kube-api-access-xz7dh\") pod \"managed-serviceaccount-addon-agent-6947cb4444-6vzp5\" (UID: \"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.239773 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfcd50fd-8aa1-4d83-a472-d949312be4a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.239773 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzsl\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-kube-api-access-7gzsl\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.239773 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed273e4-d733-4174-9881-e939528c79f2-ca-trust-extracted\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.239773 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfcd50fd-8aa1-4d83-a472-d949312be4a3-tmp\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.239773 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-registry-tls\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.240023 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed273e4-d733-4174-9881-e939528c79f2-trusted-ca\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.240023 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6947cb4444-6vzp5\" (UID: \"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.240023 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed273e4-d733-4174-9881-e939528c79f2-installation-pull-secrets\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.240023 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjct\" (UniqueName: \"kubernetes.io/projected/dfcd50fd-8aa1-4d83-a472-d949312be4a3-kube-api-access-lhjct\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.240023 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.239995 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfcd50fd-8aa1-4d83-a472-d949312be4a3-tmp\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.242180 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.242153 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6947cb4444-6vzp5\" (UID: \"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.242180 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.242162 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dfcd50fd-8aa1-4d83-a472-d949312be4a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.257245 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.257225 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjct\" (UniqueName: \"kubernetes.io/projected/dfcd50fd-8aa1-4d83-a472-d949312be4a3-kube-api-access-lhjct\") pod \"klusterlet-addon-workmgr-7c587975b9-7qbxb\" (UID: \"dfcd50fd-8aa1-4d83-a472-d949312be4a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.257688 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.257669 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7dh\" (UniqueName: \"kubernetes.io/projected/3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9-kube-api-access-xz7dh\") pod \"managed-serviceaccount-addon-agent-6947cb4444-6vzp5\" (UID: \"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.317741 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.317710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:45.317741 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.317727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzwm5" event={"ID":"1d595560-99c7-447b-9685-9e432ee1fde4","Type":"ContainerStarted","Data":"c31c070153a8a80c0d515b3c11679e628c6cac734b45c52a0df2963fac18cf92"} Jul 02 19:54:45.324580 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.324560 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:45.340847 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.340826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:45.341026 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed273e4-d733-4174-9881-e939528c79f2-installation-pull-secrets\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341091 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed273e4-d733-4174-9881-e939528c79f2-image-registry-private-configuration\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341091 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed273e4-d733-4174-9881-e939528c79f2-registry-certificates\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341190 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-bound-sa-token\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341360 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzsl\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-kube-api-access-7gzsl\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341471 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed273e4-d733-4174-9881-e939528c79f2-ca-trust-extracted\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341471 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-registry-tls\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341590 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed273e4-d733-4174-9881-e939528c79f2-trusted-ca\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341775 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341748 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ed273e4-d733-4174-9881-e939528c79f2-ca-trust-extracted\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.341984 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.341948 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ed273e4-d733-4174-9881-e939528c79f2-registry-certificates\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.342357 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.342338 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ed273e4-d733-4174-9881-e939528c79f2-trusted-ca\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.343604 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.343582 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ed273e4-d733-4174-9881-e939528c79f2-image-registry-private-configuration\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.343833 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.343815 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ed273e4-d733-4174-9881-e939528c79f2-installation-pull-secrets\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.343928 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.343904 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-registry-tls\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.349641 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.349622 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-bound-sa-token\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.350692 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.350674 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzsl\" (UniqueName: \"kubernetes.io/projected/6ed273e4-d733-4174-9881-e939528c79f2-kube-api-access-7gzsl\") pod \"image-registry-68f6c7f4bf-pvrmp\" (UID: \"6ed273e4-d733-4174-9881-e939528c79f2\") " pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.357480 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.357449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" Jul 02 19:54:45.362570 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.362532 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hzwm5" podStartSLOduration=17.363092038 podStartE2EDuration="24.362518335s" totalImagesPullingTime="6.999426297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:54:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:45.360743323 +0000 UTC m=+64.729480048" watchObservedRunningTime="2026-07-02 19:54:45.362518335 +0000 UTC m=+64.731255075" Jul 02 19:54:45.443037 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.442196 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-image-registry-private-configuration\") pod \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " Jul 02 19:54:45.443037 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.442238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de5b2d34-bd4c-4036-a4be-9b1f928a0346-ca-trust-extracted\") pod \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " Jul 02 19:54:45.443037 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.442283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-installation-pull-secrets\") pod \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " Jul 02 19:54:45.443037 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.442322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-trusted-ca\") pod \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " Jul 02 19:54:45.443037 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.442371 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-certificates\") pod \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " Jul 02 19:54:45.443037 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.442404 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-bound-sa-token\") pod \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " Jul 02 19:54:45.443037 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.442439 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhs2q\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-kube-api-access-rhs2q\") pod \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\" (UID: \"de5b2d34-bd4c-4036-a4be-9b1f928a0346\") " Jul 02 19:54:45.444373 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.443585 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5b2d34-bd4c-4036-a4be-9b1f928a0346-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "de5b2d34-bd4c-4036-a4be-9b1f928a0346" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:54:45.444373 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.443854 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "de5b2d34-bd4c-4036-a4be-9b1f928a0346" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:54:45.444373 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.444064 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "de5b2d34-bd4c-4036-a4be-9b1f928a0346" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:54:45.444686 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.444435 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-trusted-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:45.445164 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.445118 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "de5b2d34-bd4c-4036-a4be-9b1f928a0346" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:54:45.445341 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.444456 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-certificates\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:45.445341 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.445308 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de5b2d34-bd4c-4036-a4be-9b1f928a0346-ca-trust-extracted\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:45.446398 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.446354 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "de5b2d34-bd4c-4036-a4be-9b1f928a0346" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:54:45.446681 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.446635 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-kube-api-access-rhs2q" (OuterVolumeSpecName: "kube-api-access-rhs2q") pod "de5b2d34-bd4c-4036-a4be-9b1f928a0346" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346"). InnerVolumeSpecName "kube-api-access-rhs2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:54:45.455159 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.455123 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "de5b2d34-bd4c-4036-a4be-9b1f928a0346" (UID: "de5b2d34-bd4c-4036-a4be-9b1f928a0346"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:54:45.462994 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.462970 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hfqbh\"" Jul 02 19:54:45.470547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.470528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:45.481064 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.481000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb"] Jul 02 19:54:45.483153 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:45.483128 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcd50fd_8aa1_4d83_a472_d949312be4a3.slice/crio-5a3d06d8212057d0f5de6f77ede4c89de749a0d856c8c90c0fdc4cff2000b531 WatchSource:0}: Error finding container 5a3d06d8212057d0f5de6f77ede4c89de749a0d856c8c90c0fdc4cff2000b531: Status 404 returned error can't find the container with id 5a3d06d8212057d0f5de6f77ede4c89de749a0d856c8c90c0fdc4cff2000b531 Jul 02 19:54:45.496060 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.496023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5"] Jul 02 19:54:45.498574 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:45.498548 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ec39aa9_2cd8_4b6c_bd49_18d39c6f9df9.slice/crio-74a27a24df3317884f9f721f0031fd5b2dd03f92d607751ef0652b8b7105ef6f WatchSource:0}: Error finding container 74a27a24df3317884f9f721f0031fd5b2dd03f92d607751ef0652b8b7105ef6f: Status 404 returned error can't find the container with id 74a27a24df3317884f9f721f0031fd5b2dd03f92d607751ef0652b8b7105ef6f Jul 02 19:54:45.546310 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.546284 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-installation-pull-secrets\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:45.546310 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.546307 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-bound-sa-token\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:45.546508 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.546319 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhs2q\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-kube-api-access-rhs2q\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:45.546508 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.546328 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de5b2d34-bd4c-4036-a4be-9b1f928a0346-image-registry-private-configuration\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:45.601097 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:45.601063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f6c7f4bf-pvrmp"] Jul 02 19:54:45.604215 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:45.604189 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed273e4_d733_4174_9881_e939528c79f2.slice/crio-053bf06d79e936dd06b133ad04094405d8ccaf38af899b9e10565f41a76bb95e WatchSource:0}: Error finding container 053bf06d79e936dd06b133ad04094405d8ccaf38af899b9e10565f41a76bb95e: Status 404 returned error can't find the container with id 053bf06d79e936dd06b133ad04094405d8ccaf38af899b9e10565f41a76bb95e Jul 02 19:54:46.324749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.324652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" event={"ID":"6ed273e4-d733-4174-9881-e939528c79f2","Type":"ContainerStarted","Data":"bbf845838b05a23f70fcc4848335cf5ffe7db9ecec97bca3b88286ee0c34bd4c"} Jul 02 19:54:46.324749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.324722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" event={"ID":"6ed273e4-d733-4174-9881-e939528c79f2","Type":"ContainerStarted","Data":"053bf06d79e936dd06b133ad04094405d8ccaf38af899b9e10565f41a76bb95e"} Jul 02 19:54:46.325259 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.324889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:54:46.325987 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.325958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" event={"ID":"dfcd50fd-8aa1-4d83-a472-d949312be4a3","Type":"ContainerStarted","Data":"5a3d06d8212057d0f5de6f77ede4c89de749a0d856c8c90c0fdc4cff2000b531"} Jul 02 19:54:46.327567 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.327542 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" event={"ID":"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9","Type":"ContainerStarted","Data":"74a27a24df3317884f9f721f0031fd5b2dd03f92d607751ef0652b8b7105ef6f"} Jul 02 19:54:46.327671 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.327547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5984fbd7f-9tc9p" Jul 02 19:54:46.347157 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.347096 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" podStartSLOduration=1.3470780439999999 podStartE2EDuration="1.347078044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:46.345946945 +0000 UTC m=+65.714683682" watchObservedRunningTime="2026-07-02 19:54:46.347078044 +0000 UTC m=+65.715814751" Jul 02 19:54:46.375693 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.375660 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5984fbd7f-9tc9p"] Jul 02 19:54:46.379357 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.379332 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5984fbd7f-9tc9p"] Jul 02 19:54:46.455155 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.455116 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de5b2d34-bd4c-4036-a4be-9b1f928a0346-registry-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:54:46.757748 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.757707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:46.764015 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.762532 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:54:46.770608 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.770581 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48751f64-539d-4c0f-a40f-84264b0b6720-original-pull-secret\") pod \"global-pull-secret-syncer-g7p4v\" (UID: \"48751f64-539d-4c0f-a40f-84264b0b6720\") " pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:46.859173 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.859137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:46.859173 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.859180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:46.862023 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.862004 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:54:46.862063 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.862045 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:54:46.872653 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.872626 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9291347-9dc5-48e1-b2af-ea6db41aa368-metrics-certs\") pod \"network-metrics-daemon-gm4b8\" (UID: \"d9291347-9dc5-48e1-b2af-ea6db41aa368\") " pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:46.873217 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.873201 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:54:46.882749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.882726 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d874\" (UniqueName: \"kubernetes.io/projected/dea8b301-09ab-4de3-93fe-f4cc9c01dbab-kube-api-access-7d874\") pod \"network-check-target-8ngzh\" (UID: \"dea8b301-09ab-4de3-93fe-f4cc9c01dbab\") " pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:46.962241 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.962208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7p4v" Jul 02 19:54:46.972297 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.972262 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k665n\"" Jul 02 19:54:46.978173 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.978141 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdnmq\"" Jul 02 19:54:46.980248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.980225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:46.985858 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:46.985808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gm4b8" Jul 02 19:54:47.056252 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.055925 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5b2d34-bd4c-4036-a4be-9b1f928a0346" path="/var/lib/kubelet/pods/de5b2d34-bd4c-4036-a4be-9b1f928a0346/volumes" Jul 02 19:54:47.138967 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.138913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7p4v"] Jul 02 19:54:47.142130 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:47.142104 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48751f64_539d_4c0f_a40f_84264b0b6720.slice/crio-3da16d4ce05cc5b09235225c8b8524efabb42052b3b6fd95b2b3b3db3bbed4a2 WatchSource:0}: Error finding container 3da16d4ce05cc5b09235225c8b8524efabb42052b3b6fd95b2b3b3db3bbed4a2: Status 404 returned error can't find the container with id 3da16d4ce05cc5b09235225c8b8524efabb42052b3b6fd95b2b3b3db3bbed4a2 Jul 02 19:54:47.157758 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.157733 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8ngzh"] Jul 02 19:54:47.182402 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.182371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gm4b8"] Jul 02 19:54:47.333068 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.332982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7p4v" event={"ID":"48751f64-539d-4c0f-a40f-84264b0b6720","Type":"ContainerStarted","Data":"3da16d4ce05cc5b09235225c8b8524efabb42052b3b6fd95b2b3b3db3bbed4a2"} Jul 02 19:54:47.464456 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.464416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:47.467130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.467103 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/856c4e6a-55d7-414e-b534-a6d36dc652f3-metrics-tls\") pod \"dns-default-w6qq7\" (UID: \"856c4e6a-55d7-414e-b534-a6d36dc652f3\") " pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:47.652453 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.652420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nzdvh\"" Jul 02 19:54:47.659862 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:47.659840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:47.930618 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:47.930539 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9291347_9dc5_48e1_b2af_ea6db41aa368.slice/crio-cb88f7ec1fa225ca108fb3e4ee0f1558e6485d93f734b7986e57ce6d2038762f WatchSource:0}: Error finding container cb88f7ec1fa225ca108fb3e4ee0f1558e6485d93f734b7986e57ce6d2038762f: Status 404 returned error can't find the container with id cb88f7ec1fa225ca108fb3e4ee0f1558e6485d93f734b7986e57ce6d2038762f Jul 02 19:54:47.931202 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:47.931180 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea8b301_09ab_4de3_93fe_f4cc9c01dbab.slice/crio-6da70fed2f51118d08efab40ff44019c4a398b26da946ff3c520808843496d9e WatchSource:0}: Error finding container 6da70fed2f51118d08efab40ff44019c4a398b26da946ff3c520808843496d9e: Status 404 returned error can't find the container with id 6da70fed2f51118d08efab40ff44019c4a398b26da946ff3c520808843496d9e Jul 02 19:54:48.337701 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:48.337616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gm4b8" event={"ID":"d9291347-9dc5-48e1-b2af-ea6db41aa368","Type":"ContainerStarted","Data":"cb88f7ec1fa225ca108fb3e4ee0f1558e6485d93f734b7986e57ce6d2038762f"} Jul 02 19:54:48.338960 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:48.338915 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8ngzh" event={"ID":"dea8b301-09ab-4de3-93fe-f4cc9c01dbab","Type":"ContainerStarted","Data":"6da70fed2f51118d08efab40ff44019c4a398b26da946ff3c520808843496d9e"} Jul 02 19:54:48.518410 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:48.518374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w6qq7"] Jul 02 19:54:48.521654 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:48.521625 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856c4e6a_55d7_414e_b534_a6d36dc652f3.slice/crio-1833d70f91f4532e26b245b6b6fc059a4661d123b82106197f6e04c6e2804400 WatchSource:0}: Error finding container 1833d70f91f4532e26b245b6b6fc059a4661d123b82106197f6e04c6e2804400: Status 404 returned error can't find the container with id 1833d70f91f4532e26b245b6b6fc059a4661d123b82106197f6e04c6e2804400 Jul 02 19:54:49.345966 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:49.345925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" event={"ID":"3ec39aa9-2cd8-4b6c-bd49-18d39c6f9df9","Type":"ContainerStarted","Data":"df6a62431e8452703c7b736feb2c91ca33ac2faa6fdd4c0038a8100dcbf0a1be"} Jul 02 19:54:49.348180 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:49.348135 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6qq7" event={"ID":"856c4e6a-55d7-414e-b534-a6d36dc652f3","Type":"ContainerStarted","Data":"1833d70f91f4532e26b245b6b6fc059a4661d123b82106197f6e04c6e2804400"} Jul 02 19:54:49.363560 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:49.363504 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6947cb4444-6vzp5" podStartSLOduration=1.105386342 podStartE2EDuration="4.363483051s" totalImagesPullingTime="3.258096709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:49.362740941 +0000 UTC m=+68.731477667" watchObservedRunningTime="2026-07-02 19:54:49.363483051 +0000 UTC m=+68.732219778" Jul 02 19:54:50.354926 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:50.354856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gm4b8" event={"ID":"d9291347-9dc5-48e1-b2af-ea6db41aa368","Type":"ContainerStarted","Data":"f12408fdebaa37685b3bc36ca0e2c29196513889e46ebb565e3066ad21f26283"} Jul 02 19:54:53.142360 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.142316 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk"] Jul 02 19:54:53.146310 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.146285 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.149396 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.149372 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 19:54:53.150347 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.150323 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 19:54:53.150449 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.150345 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hbr6x\"" Jul 02 19:54:53.151077 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.150599 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 19:54:53.151077 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.150781 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 19:54:53.151077 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.150920 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 19:54:53.156480 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.155326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk"] Jul 02 19:54:53.214105 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.214076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.214221 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.214124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.214221 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.214199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9bffdf5-591b-4022-9d16-78a62be3a0ff-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.214313 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.214253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bx8\" (UniqueName: \"kubernetes.io/projected/a9bffdf5-591b-4022-9d16-78a62be3a0ff-kube-api-access-76bx8\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.315071 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.315029 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.315241 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.315083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.315241 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.315120 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9bffdf5-591b-4022-9d16-78a62be3a0ff-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.315241 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.315147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bx8\" (UniqueName: \"kubernetes.io/projected/a9bffdf5-591b-4022-9d16-78a62be3a0ff-kube-api-access-76bx8\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.315241 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:53.315200 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 02 19:54:53.315435 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:53.315280 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-tls podName:a9bffdf5-591b-4022-9d16-78a62be3a0ff nodeName:}" failed. No retries permitted until 2026-07-02 19:54:53.815257832 +0000 UTC m=+73.183994541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-w4wqk" (UID: "a9bffdf5-591b-4022-9d16-78a62be3a0ff") : secret "prometheus-operator-tls" not found Jul 02 19:54:53.315917 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.315894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9bffdf5-591b-4022-9d16-78a62be3a0ff-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.317484 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.317450 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.323615 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.323596 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bx8\" (UniqueName: \"kubernetes.io/projected/a9bffdf5-591b-4022-9d16-78a62be3a0ff-kube-api-access-76bx8\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.370557 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.370514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8ngzh" event={"ID":"dea8b301-09ab-4de3-93fe-f4cc9c01dbab","Type":"ContainerStarted","Data":"576ec3b3ba34231adce60fca88211b186c521768c39e8cc0c5b701a273832760"} Jul 02 19:54:53.370744 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.370605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:54:53.372196 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.372167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6qq7" event={"ID":"856c4e6a-55d7-414e-b534-a6d36dc652f3","Type":"ContainerStarted","Data":"a7586235e05d56ee557659b84498878408ed6c2e3d96d36c715ae95c5b616e19"} Jul 02 19:54:53.372317 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.372216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w6qq7" event={"ID":"856c4e6a-55d7-414e-b534-a6d36dc652f3","Type":"ContainerStarted","Data":"30bb497945805c4e3bdede7efa5073ce5b7d2f3ce199af593ec576f143bedec8"} Jul 02 19:54:53.372317 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.372246 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w6qq7" Jul 02 19:54:53.373901 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.373876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gm4b8" event={"ID":"d9291347-9dc5-48e1-b2af-ea6db41aa368","Type":"ContainerStarted","Data":"ba0d7d926219d84a6063b8eefb75d83d184567f9fd98b0e4b387374758c3b303"} Jul 02 19:54:53.375215 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.375191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7p4v" event={"ID":"48751f64-539d-4c0f-a40f-84264b0b6720","Type":"ContainerStarted","Data":"5cd70f8d73e47b845334b23157c671f067709671a89ec503bb86a96540ab5d4a"} Jul 02 19:54:53.387113 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.387074 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8ngzh" podStartSLOduration=67.326732898 podStartE2EDuration="1m12.387063772s" totalImagesPullingTime="5.060330874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:53.386305421 +0000 UTC m=+72.755042145" watchObservedRunningTime="2026-07-02 19:54:53.387063772 +0000 UTC m=+72.755800497" Jul 02 19:54:53.403571 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.403532 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g7p4v" podStartSLOduration=66.555093605 podStartE2EDuration="1m12.403519878s" totalImagesPullingTime="5.848426273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:53.402873663 +0000 UTC m=+72.771610390" watchObservedRunningTime="2026-07-02 19:54:53.403519878 +0000 UTC m=+72.772256604" Jul 02 19:54:53.418921 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.418883 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gm4b8" podStartSLOduration=70.513557616 podStartE2EDuration="1m12.418870667s" totalImagesPullingTime="1.905313051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:53.418291278 +0000 UTC m=+72.787028003" watchObservedRunningTime="2026-07-02 19:54:53.418870667 +0000 UTC m=+72.787607391" Jul 02 19:54:53.436719 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.436641 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w6qq7" podStartSLOduration=33.97688837 podStartE2EDuration="38.436631444s" totalImagesPullingTime="4.459743074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:53.435675303 +0000 UTC m=+72.804412018" watchObservedRunningTime="2026-07-02 19:54:53.436631444 +0000 UTC m=+72.805368169" Jul 02 19:54:53.820715 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.820612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:53.822969 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:53.822941 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bffdf5-591b-4022-9d16-78a62be3a0ff-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4wqk\" (UID: \"a9bffdf5-591b-4022-9d16-78a62be3a0ff\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:54.060504 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:54.060446 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" Jul 02 19:54:54.182203 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:54.182178 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk"] Jul 02 19:54:54.184612 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:54.184582 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bffdf5_591b_4022_9d16_78a62be3a0ff.slice/crio-7fa45159c576c70aaecaaa1851cd82b984e2242c09b924d01b856fa8a1974706 WatchSource:0}: Error finding container 7fa45159c576c70aaecaaa1851cd82b984e2242c09b924d01b856fa8a1974706: Status 404 returned error can't find the container with id 7fa45159c576c70aaecaaa1851cd82b984e2242c09b924d01b856fa8a1974706 Jul 02 19:54:54.379921 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:54.379826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" event={"ID":"a9bffdf5-591b-4022-9d16-78a62be3a0ff","Type":"ContainerStarted","Data":"7fa45159c576c70aaecaaa1851cd82b984e2242c09b924d01b856fa8a1974706"} Jul 02 19:54:56.389441 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:56.389405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" event={"ID":"a9bffdf5-591b-4022-9d16-78a62be3a0ff","Type":"ContainerStarted","Data":"8ece82b6485c56d0d63320335f28b73c1ed696d530b36cab6e2eb99cccf3dff3"} Jul 02 19:54:56.389947 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:56.389479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" event={"ID":"a9bffdf5-591b-4022-9d16-78a62be3a0ff","Type":"ContainerStarted","Data":"98e38538669cfb86f5313d334aa8969ddab8b2aeba218b2bc7f717affda5fd38"} Jul 02 19:54:56.409038 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:56.408987 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4wqk" podStartSLOduration=1.84261531 podStartE2EDuration="3.408967092s" totalImagesPullingTime="1.566351782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:56.407008298 +0000 UTC m=+75.775745023" watchObservedRunningTime="2026-07-02 19:54:56.408967092 +0000 UTC m=+75.777703818" Jul 02 19:54:58.516318 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.516098 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k57rk"] Jul 02 19:54:58.521621 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.521589 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9"] Jul 02 19:54:58.522945 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.522920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.525523 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.525501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.526388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.526194 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 19:54:58.526388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.526253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56bzk\"" Jul 02 19:54:58.526388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.526304 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 19:54:58.526388 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.526194 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 19:54:58.528226 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.528207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 02 19:54:58.528405 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.528386 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 02 19:54:58.528879 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.528861 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x8tx2\"" Jul 02 19:54:58.528979 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.528939 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 02 19:54:58.535954 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.535850 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9"] Jul 02 19:54:58.662976 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.662950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.663100 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.662987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfz2\" (UniqueName: \"kubernetes.io/projected/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-api-access-sbfz2\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.663100 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcec82c2-f134-4207-9604-bc21397bd910-metrics-client-ca\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663100 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-wtmp\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663100 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-tls\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663279 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-accelerators-collector-config\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663279 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-root\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663279 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-textfile\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663279 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.663495 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-sys\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663495 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.663495 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgd4j\" (UniqueName: \"kubernetes.io/projected/bcec82c2-f134-4207-9604-bc21397bd910-kube-api-access-kgd4j\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.663495 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29116bb4-6730-41f5-9c78-7b7efe438cfe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.663661 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29116bb4-6730-41f5-9c78-7b7efe438cfe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.663661 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.663530 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764061 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-sys\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764226 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.764226 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgd4j\" (UniqueName: \"kubernetes.io/projected/bcec82c2-f134-4207-9604-bc21397bd910-kube-api-access-kgd4j\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764226 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764162 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-sys\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764226 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29116bb4-6730-41f5-9c78-7b7efe438cfe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.764444 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29116bb4-6730-41f5-9c78-7b7efe438cfe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.764444 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764444 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.764444 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfz2\" (UniqueName: \"kubernetes.io/projected/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-api-access-sbfz2\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.764444 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcec82c2-f134-4207-9604-bc21397bd910-metrics-client-ca\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764444 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-wtmp\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-tls\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-accelerators-collector-config\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764555 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29116bb4-6730-41f5-9c78-7b7efe438cfe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.764749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-root\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-textfile\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.764749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.764641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.765072 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:58.764894 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 02 19:54:58.765072 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:58.764981 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-tls podName:bcec82c2-f134-4207-9604-bc21397bd910 nodeName:}" failed. No retries permitted until 2026-07-02 19:54:59.264959541 +0000 UTC m=+78.633696258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-tls") pod "node-exporter-k57rk" (UID: "bcec82c2-f134-4207-9604-bc21397bd910") : secret "node-exporter-tls" not found Jul 02 19:54:58.765948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.765280 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-wtmp\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.765948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.765352 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcec82c2-f134-4207-9604-bc21397bd910-root\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.765948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.765390 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcec82c2-f134-4207-9604-bc21397bd910-metrics-client-ca\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.765948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.765419 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.765948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.765697 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-textfile\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.765948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.765893 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-accelerators-collector-config\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.766418 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.766351 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29116bb4-6730-41f5-9c78-7b7efe438cfe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.768199 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.768177 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.768504 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.768483 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.769161 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.769143 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.776448 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.773992 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfz2\" (UniqueName: \"kubernetes.io/projected/29116bb4-6730-41f5-9c78-7b7efe438cfe-kube-api-access-sbfz2\") pod \"kube-state-metrics-7c9dbb8b7d-4q4s9\" (UID: \"29116bb4-6730-41f5-9c78-7b7efe438cfe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:58.776448 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.774269 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgd4j\" (UniqueName: \"kubernetes.io/projected/bcec82c2-f134-4207-9604-bc21397bd910-kube-api-access-kgd4j\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:58.845144 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:58.845100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" Jul 02 19:54:59.013978 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.013950 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9"] Jul 02 19:54:59.017848 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:59.017776 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29116bb4_6730_41f5_9c78_7b7efe438cfe.slice/crio-4b13d8907690a2841fd339bd5ca8bfc25cbf9f0219c98cd37f1c17713d7edc5c WatchSource:0}: Error finding container 4b13d8907690a2841fd339bd5ca8bfc25cbf9f0219c98cd37f1c17713d7edc5c: Status 404 returned error can't find the container with id 4b13d8907690a2841fd339bd5ca8bfc25cbf9f0219c98cd37f1c17713d7edc5c Jul 02 19:54:59.268822 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.268736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-tls\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:59.271055 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.271027 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcec82c2-f134-4207-9604-bc21397bd910-node-exporter-tls\") pod \"node-exporter-k57rk\" (UID: \"bcec82c2-f134-4207-9604-bc21397bd910\") " pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:59.406478 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.406424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" event={"ID":"dfcd50fd-8aa1-4d83-a472-d949312be4a3","Type":"ContainerStarted","Data":"bf8ae6a9d60c58ba0e2ed3d4a1a7dcde6b1fbe7be67183f10091674a38d30fbd"} Jul 02 19:54:59.406642 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.406589 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:59.407868 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.407802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" event={"ID":"29116bb4-6730-41f5-9c78-7b7efe438cfe","Type":"ContainerStarted","Data":"4b13d8907690a2841fd339bd5ca8bfc25cbf9f0219c98cd37f1c17713d7edc5c"} Jul 02 19:54:59.408631 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.408610 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" Jul 02 19:54:59.427025 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.426978 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c587975b9-7qbxb" podStartSLOduration=1.310189361 podStartE2EDuration="14.426964714s" totalImagesPullingTime="13.116775353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:54:59.42541267 +0000 UTC m=+78.794149397" watchObservedRunningTime="2026-07-02 19:54:59.426964714 +0000 UTC m=+78.795701439" Jul 02 19:54:59.437927 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.437903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k57rk" Jul 02 19:54:59.450529 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:54:59.450497 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcec82c2_f134_4207_9604_bc21397bd910.slice/crio-7a3c9a32bcfd1b3f31d2d79bd475ba838755067648c83adc242c8af8833f4c12 WatchSource:0}: Error finding container 7a3c9a32bcfd1b3f31d2d79bd475ba838755067648c83adc242c8af8833f4c12: Status 404 returned error can't find the container with id 7a3c9a32bcfd1b3f31d2d79bd475ba838755067648c83adc242c8af8833f4c12 Jul 02 19:54:59.587529 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.587438 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:54:59.591317 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.591274 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.594616 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.594387 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 19:54:59.594616 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.594406 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 19:54:59.594616 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.594407 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 19:54:59.594616 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.594391 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 19:54:59.594960 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.594652 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 19:54:59.594960 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.594680 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 19:54:59.595072 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.594980 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k7jz5\"" Jul 02 19:54:59.595169 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.595153 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 19:54:59.595706 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.595685 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 19:54:59.595822 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.595685 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 19:54:59.605721 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.605638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:54:59.671820 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.671790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.671983 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.671828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.671983 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.671852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.671983 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.671934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6gg\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-kube-api-access-dq6gg\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.671983 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.671970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672197 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.671995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-out\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672197 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.672035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-web-config\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672197 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.672081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672197 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.672118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672197 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.672152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.672213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.672250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.672438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.672270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773659 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773659 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773898 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773898 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773898 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773898 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773898 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6gg\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-kube-api-access-dq6gg\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.773898 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.773812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774203 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.774119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-out\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774203 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.774150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-web-config\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774203 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.774198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774349 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.774216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774349 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.774257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774349 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.774282 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774874 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.774839 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.774983 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:59.774962 2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 02 19:54:59.775048 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.775000 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.775048 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:54:59.775026 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls podName:b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3 nodeName:}" failed. No retries permitted until 2026-07-02 19:55:00.275006425 +0000 UTC m=+79.643743144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3") : secret "alertmanager-main-tls" not found Jul 02 19:54:59.776716 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.776668 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.777080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.777054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.777569 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.777544 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.777663 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.777614 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.777774 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.777749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.778071 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.778049 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-web-config\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.778147 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.778092 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.778838 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.778820 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-out\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:54:59.782017 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:54:59.781999 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6gg\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-kube-api-access-dq6gg\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:55:00.280036 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:00.279991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:55:00.283659 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:00.283630 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:55:00.414636 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:00.414603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k57rk" event={"ID":"bcec82c2-f134-4207-9604-bc21397bd910","Type":"ContainerStarted","Data":"7a3c9a32bcfd1b3f31d2d79bd475ba838755067648c83adc242c8af8833f4c12"} Jul 02 19:55:00.506339 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:00.506302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:55:00.841750 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:00.841713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:55:00.844336 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:55:00.844294 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83bb63a_e34a_4a7a_9838_a3adc1c5f0d3.slice/crio-d4f7c4f8e262dc5a8d401d00be341e8ca76ede2be825cab16fe5aa27a98f99fe WatchSource:0}: Error finding container d4f7c4f8e262dc5a8d401d00be341e8ca76ede2be825cab16fe5aa27a98f99fe: Status 404 returned error can't find the container with id d4f7c4f8e262dc5a8d401d00be341e8ca76ede2be825cab16fe5aa27a98f99fe Jul 02 19:55:01.419925 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:01.419889 2564 generic.go:356] "Generic (PLEG): container finished" podID="bcec82c2-f134-4207-9604-bc21397bd910" containerID="73eb40c691fae887b56decf85e7f96cd9fe1ef2719e04b761ac5058f9942b76c" exitCode=0 Jul 02 19:55:01.420077 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:01.419980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k57rk" event={"ID":"bcec82c2-f134-4207-9604-bc21397bd910","Type":"ContainerDied","Data":"73eb40c691fae887b56decf85e7f96cd9fe1ef2719e04b761ac5058f9942b76c"} Jul 02 19:55:01.422246 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:01.422217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" event={"ID":"29116bb4-6730-41f5-9c78-7b7efe438cfe","Type":"ContainerStarted","Data":"01effe3eb9fd2a63be008db66a5c0ddc71095ed795627ed41d8f4fbc25d799c5"} Jul 02 19:55:01.422381 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:01.422277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" event={"ID":"29116bb4-6730-41f5-9c78-7b7efe438cfe","Type":"ContainerStarted","Data":"35a8ee7b7bfe14b37faf0ee6cda4486a12f6fc7bc45560a63f526f414fc66807"} Jul 02 19:55:01.422381 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:01.422302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" event={"ID":"29116bb4-6730-41f5-9c78-7b7efe438cfe","Type":"ContainerStarted","Data":"6fd3bd94a09fd79f07f8aad66e179f2149b4f60685059017e895f346cf7b4a3c"} Jul 02 19:55:01.423648 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:01.423625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerStarted","Data":"d4f7c4f8e262dc5a8d401d00be341e8ca76ede2be825cab16fe5aa27a98f99fe"} Jul 02 19:55:01.467100 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:01.467045 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4q4s9" podStartSLOduration=1.78749511 podStartE2EDuration="3.467029524s" totalImagesPullingTime="1.679534414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:55:01.467008495 +0000 UTC m=+80.835745221" watchObservedRunningTime="2026-07-02 19:55:01.467029524 +0000 UTC m=+80.835766249" Jul 02 19:55:02.430562 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:02.430524 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k57rk" event={"ID":"bcec82c2-f134-4207-9604-bc21397bd910","Type":"ContainerStarted","Data":"e275ccd1d4d67415c4d36927ad7762eaf0d69f9de30fed9f1e78419412f102ba"} Jul 02 19:55:02.431023 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:02.430606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k57rk" event={"ID":"bcec82c2-f134-4207-9604-bc21397bd910","Type":"ContainerStarted","Data":"f6d3ed6703f0ce4d8237d50de66ac3785f321b2a0c222b85f03b397fc7844f19"} Jul 02 19:55:02.453218 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:02.453169 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k57rk" podStartSLOduration=3.106696595 podStartE2EDuration="4.453150499s" totalImagesPullingTime="1.247680979s" totalInitContainerRuntime="98.772925ms" isStatefulPod=true podCreationTimestamp="2026-07-02 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:55:02.452044613 +0000 UTC m=+81.820781339" watchObservedRunningTime="2026-07-02 19:55:02.453150499 +0000 UTC m=+81.821887223" Jul 02 19:55:03.279146 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.279109 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr"] Jul 02 19:55:03.282670 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.282651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.285283 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.285261 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-58w5t\"" Jul 02 19:55:03.285390 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.285284 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 02 19:55:03.289646 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.289598 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr"] Jul 02 19:55:03.382413 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.382387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w6qq7" Jul 02 19:55:03.408539 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.408507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kkv\" (UniqueName: \"kubernetes.io/projected/40fda62b-664c-4637-a42d-acbc0f56657d-kube-api-access-44kkv\") pod \"monitoring-plugin-d5dcff7cf-7lhrr\" (UID: \"40fda62b-664c-4637-a42d-acbc0f56657d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.408539 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.408541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40fda62b-664c-4637-a42d-acbc0f56657d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-7lhrr\" (UID: \"40fda62b-664c-4637-a42d-acbc0f56657d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.436130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.436101 2564 generic.go:356] "Generic (PLEG): container finished" podID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerID="5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a" exitCode=0 Jul 02 19:55:03.436545 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.436194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a"} Jul 02 19:55:03.509563 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.509532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kkv\" (UniqueName: \"kubernetes.io/projected/40fda62b-664c-4637-a42d-acbc0f56657d-kube-api-access-44kkv\") pod \"monitoring-plugin-d5dcff7cf-7lhrr\" (UID: \"40fda62b-664c-4637-a42d-acbc0f56657d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.509749 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.509572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40fda62b-664c-4637-a42d-acbc0f56657d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-7lhrr\" (UID: \"40fda62b-664c-4637-a42d-acbc0f56657d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.511988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.511968 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40fda62b-664c-4637-a42d-acbc0f56657d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-7lhrr\" (UID: \"40fda62b-664c-4637-a42d-acbc0f56657d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.518201 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.518182 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kkv\" (UniqueName: \"kubernetes.io/projected/40fda62b-664c-4637-a42d-acbc0f56657d-kube-api-access-44kkv\") pod \"monitoring-plugin-d5dcff7cf-7lhrr\" (UID: \"40fda62b-664c-4637-a42d-acbc0f56657d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.592888 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.592801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:03.717390 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:03.717237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr"] Jul 02 19:55:03.720164 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:55:03.720139 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fda62b_664c_4637_a42d_acbc0f56657d.slice/crio-ed49424276f57ffc50fae6f9cdd9e59f4a297cd24fa325346df95e3120e59cef WatchSource:0}: Error finding container ed49424276f57ffc50fae6f9cdd9e59f4a297cd24fa325346df95e3120e59cef: Status 404 returned error can't find the container with id ed49424276f57ffc50fae6f9cdd9e59f4a297cd24fa325346df95e3120e59cef Jul 02 19:55:04.445107 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.445063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" event={"ID":"40fda62b-664c-4637-a42d-acbc0f56657d","Type":"ContainerStarted","Data":"ed49424276f57ffc50fae6f9cdd9e59f4a297cd24fa325346df95e3120e59cef"} Jul 02 19:55:04.791499 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.790625 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:55:04.796350 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.796323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.799529 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.799207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 02 19:55:04.799529 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.799400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 02 19:55:04.799529 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.799413 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 02 19:55:04.799764 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.799691 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 02 19:55:04.799926 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.799906 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 19:55:04.799990 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.799957 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 02 19:55:04.799990 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.799906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 02 19:55:04.800705 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.800546 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 02 19:55:04.802121 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.802071 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-vsbcmsnpq1dg\"" Jul 02 19:55:04.802121 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.802089 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 02 19:55:04.802454 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.802435 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 02 19:55:04.802538 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.802512 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 02 19:55:04.802590 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.802572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 02 19:55:04.802718 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.802666 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d55rw\"" Jul 02 19:55:04.802971 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.802955 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 02 19:55:04.804399 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.804377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 02 19:55:04.806265 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.806244 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 02 19:55:04.813595 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.813574 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:55:04.921375 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921375 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-config\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-web-config\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921610 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-config-out\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.921883 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.922454 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.922454 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:04.922454 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:04.921963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546zf\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-kube-api-access-546zf\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.022917 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.022866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.022927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.022965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.022995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-config\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-web-config\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-config-out\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023335 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.023547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546zf\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-kube-api-access-546zf\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.024254 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023893 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.024254 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.023897 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.026520 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.026488 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.026737 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.026638 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.027598 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.027566 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.028167 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.028140 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.028319 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.028295 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.028956 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.028832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-web-config\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.028956 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.028889 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.031741 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.031229 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.031741 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.031448 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.031741 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.031699 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-config\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.032543 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.031974 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-config-out\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.032543 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.032186 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.032543 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.032482 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.033299 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.032890 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.033299 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.033135 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.033477 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.033301 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-546zf\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-kube-api-access-546zf\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.033599 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.033577 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:05.114847 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:05.114769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:06.005844 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.005767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:55:06.007426 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:55:06.007400 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362f5208_8310_4e86_9736_2f4b25bd73da.slice/crio-22df597d299fb10bde5647b9a66d00b22e6299d87c72f3c8938b2a230399273a WatchSource:0}: Error finding container 22df597d299fb10bde5647b9a66d00b22e6299d87c72f3c8938b2a230399273a: Status 404 returned error can't find the container with id 22df597d299fb10bde5647b9a66d00b22e6299d87c72f3c8938b2a230399273a Jul 02 19:55:06.460554 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.460520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerStarted","Data":"2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875"} Jul 02 19:55:06.460733 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.460616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerStarted","Data":"9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4"} Jul 02 19:55:06.460733 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.460644 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerStarted","Data":"2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113"} Jul 02 19:55:06.460733 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.460674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerStarted","Data":"438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337"} Jul 02 19:55:06.460733 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.460697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerStarted","Data":"cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428"} Jul 02 19:55:06.461909 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.461883 2564 generic.go:356] "Generic (PLEG): container finished" podID="362f5208-8310-4e86-9736-2f4b25bd73da" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" exitCode=0 Jul 02 19:55:06.462001 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.461966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472"} Jul 02 19:55:06.462050 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.462036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerStarted","Data":"22df597d299fb10bde5647b9a66d00b22e6299d87c72f3c8938b2a230399273a"} Jul 02 19:55:06.463525 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.463503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" event={"ID":"40fda62b-664c-4637-a42d-acbc0f56657d","Type":"ContainerStarted","Data":"bf792243b328c8a71343d69ab78f7e44261b4fc0eeb1fbb40938017790b809f7"} Jul 02 19:55:06.463698 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.463668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:06.468195 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.468175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" Jul 02 19:55:06.502430 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:06.502384 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-7lhrr" podStartSLOduration=1.360385836 podStartE2EDuration="3.502368756s" totalImagesPullingTime="2.14198292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:55:06.501385138 +0000 UTC m=+85.870121876" watchObservedRunningTime="2026-07-02 19:55:06.502368756 +0000 UTC m=+85.871105480" Jul 02 19:55:07.338316 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:07.338241 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68f6c7f4bf-pvrmp" Jul 02 19:55:07.472907 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:07.472870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerStarted","Data":"653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3"} Jul 02 19:55:07.501437 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:07.501378 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.500733424 podStartE2EDuration="8.50135989s" totalImagesPullingTime="4.983871265s" totalInitContainerRuntime="16.755201ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:54:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:55:07.49883461 +0000 UTC m=+86.867571335" watchObservedRunningTime="2026-07-02 19:55:07.50135989 +0000 UTC m=+86.870096614" Jul 02 19:55:10.491803 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:10.491770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerStarted","Data":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} Jul 02 19:55:10.492105 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:10.491861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerStarted","Data":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} Jul 02 19:55:13.508823 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:13.508790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerStarted","Data":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} Jul 02 19:55:13.509195 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:13.508855 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerStarted","Data":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} Jul 02 19:55:13.509195 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:13.508881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerStarted","Data":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} Jul 02 19:55:13.509195 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:13.508909 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerStarted","Data":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} Jul 02 19:55:13.540132 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:13.540070 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.192153943 podStartE2EDuration="9.540052229s" totalImagesPullingTime="6.318276089s" totalInitContainerRuntime="29.622197ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:55:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:55:13.538602285 +0000 UTC m=+92.907339011" watchObservedRunningTime="2026-07-02 19:55:13.540052229 +0000 UTC m=+92.908788955" Jul 02 19:55:15.115213 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:15.115181 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:55:24.382485 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:55:24.382442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8ngzh" Jul 02 19:56:05.115294 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:05.115258 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:05.133989 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:05.133966 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:05.712203 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:05.712174 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:18.882992 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:18.882955 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:56:18.883594 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:18.883365 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="alertmanager" containerID="cri-o://cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428" gracePeriod=120 Jul 02 19:56:18.883594 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:18.883442 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy" containerID="cri-o://9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4" gracePeriod=120 Jul 02 19:56:18.883594 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:18.883514 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="config-reloader" containerID="cri-o://438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337" gracePeriod=120 Jul 02 19:56:18.883594 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:18.883429 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-web" containerID="cri-o://2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113" gracePeriod=120 Jul 02 19:56:18.883779 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:18.883429 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-metric" containerID="cri-o://2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875" gracePeriod=120 Jul 02 19:56:18.883779 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:18.883525 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="prom-label-proxy" containerID="cri-o://653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3" gracePeriod=120 Jul 02 19:56:19.760611 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760576 2564 generic.go:356] "Generic (PLEG): container finished" podID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerID="653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3" exitCode=0 Jul 02 19:56:19.760611 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760603 2564 generic.go:356] "Generic (PLEG): container finished" podID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerID="9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4" exitCode=0 Jul 02 19:56:19.760611 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760610 2564 generic.go:356] "Generic (PLEG): container finished" podID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerID="2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113" exitCode=0 Jul 02 19:56:19.760611 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760617 2564 generic.go:356] "Generic (PLEG): container finished" podID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerID="438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337" exitCode=0 Jul 02 19:56:19.760853 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760625 2564 generic.go:356] "Generic (PLEG): container finished" podID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerID="cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428" exitCode=0 Jul 02 19:56:19.760853 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3"} Jul 02 19:56:19.760853 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4"} Jul 02 19:56:19.760853 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113"} Jul 02 19:56:19.760853 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337"} Jul 02 19:56:19.760853 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:19.760796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428"} Jul 02 19:56:20.128172 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.128146 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:56:20.200020 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.199987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-metrics-client-ca\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200188 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200035 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-main-db\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200188 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200086 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-out\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200188 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-web\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200188 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200148 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-cluster-tls-config\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200188 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200179 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200446 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200207 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200446 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200446 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200292 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-web-config\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200446 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200328 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-volume\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200446 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200354 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-tls-assets\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200446 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq6gg\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-kube-api-access-dq6gg\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200446 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200421 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-trusted-ca-bundle\") pod \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\" (UID: \"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3\") " Jul 02 19:56:20.200820 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200417 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:20.200820 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.200449 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:56:20.203020 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.202975 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:20.203949 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.203921 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:20.203949 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.203933 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-kube-api-access-dq6gg" (OuterVolumeSpecName: "kube-api-access-dq6gg") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "kube-api-access-dq6gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:56:20.204064 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.203968 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:20.204162 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.204145 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:20.204394 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.204376 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:56:20.204735 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.204714 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-out" (OuterVolumeSpecName: "config-out") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:56:20.204843 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.204782 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:20.206960 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.206943 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-volume" (OuterVolumeSpecName: "config-volume") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:20.208663 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.208638 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:20.215988 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.215936 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-web-config" (OuterVolumeSpecName: "web-config") pod "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" (UID: "b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:20.301878 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301838 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-main-db\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.301878 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301869 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-out\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.301878 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301886 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301899 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-cluster-tls-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301911 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301924 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301936 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-secret-alertmanager-main-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301971 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-web-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301984 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-config-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.301994 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-tls-assets\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.302007 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6gg\" (UniqueName: \"kubernetes.io/projected/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-kube-api-access-dq6gg\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.302019 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.302130 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.302031 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3-metrics-client-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:20.767097 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.767063 2564 generic.go:356] "Generic (PLEG): container finished" podID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerID="2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875" exitCode=0 Jul 02 19:56:20.767264 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.767143 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:56:20.767264 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.767140 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875"} Jul 02 19:56:20.767264 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.767206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3","Type":"ContainerDied","Data":"d4f7c4f8e262dc5a8d401d00be341e8ca76ede2be825cab16fe5aa27a98f99fe"} Jul 02 19:56:20.767264 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.767237 2564 scope.go:117] "RemoveContainer" containerID="653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3" Jul 02 19:56:20.775929 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.775914 2564 scope.go:117] "RemoveContainer" containerID="2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875" Jul 02 19:56:20.785521 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.785500 2564 scope.go:117] "RemoveContainer" containerID="9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4" Jul 02 19:56:20.791773 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.791750 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:56:20.794483 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.794445 2564 scope.go:117] "RemoveContainer" containerID="2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113" Jul 02 19:56:20.795892 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.795871 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:56:20.800960 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.800941 2564 scope.go:117] "RemoveContainer" containerID="438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337" Jul 02 19:56:20.807646 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.807630 2564 scope.go:117] "RemoveContainer" containerID="cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428" Jul 02 19:56:20.814397 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.814379 2564 scope.go:117] "RemoveContainer" containerID="5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a" Jul 02 19:56:20.821187 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.821171 2564 scope.go:117] "RemoveContainer" containerID="653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3" Jul 02 19:56:20.821411 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:20.821393 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3\": container with ID starting with 653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3 not found: ID does not exist" containerID="653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3" Jul 02 19:56:20.821476 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.821419 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3"} err="failed to get container status \"653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3\": rpc error: code = NotFound desc = could not find container \"653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3\": container with ID starting with 653a7734f5875065ca062c5d712b8e63bf0b71ea4718905cc3a84dab2c9438c3 not found: ID does not exist" Jul 02 19:56:20.821476 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.821452 2564 scope.go:117] "RemoveContainer" containerID="2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875" Jul 02 19:56:20.821657 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:20.821636 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875\": container with ID starting with 2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875 not found: ID does not exist" containerID="2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875" Jul 02 19:56:20.821699 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.821667 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875"} err="failed to get container status \"2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875\": rpc error: code = NotFound desc = could not find container \"2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875\": container with ID starting with 2d38552abbba186f7e67779404a598004a16383d9c435eeb7787ad628c28f875 not found: ID does not exist" Jul 02 19:56:20.821699 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.821685 2564 scope.go:117] "RemoveContainer" containerID="9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4" Jul 02 19:56:20.821910 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:20.821891 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4\": container with ID starting with 9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4 not found: ID does not exist" containerID="9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4" Jul 02 19:56:20.821962 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.821914 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4"} err="failed to get container status \"9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4\": rpc error: code = NotFound desc = could not find container \"9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4\": container with ID starting with 9a4e728737c85de66d94fd56f36d47583cc601992c86513233755d873bfab5b4 not found: ID does not exist" Jul 02 19:56:20.821962 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.821927 2564 scope.go:117] "RemoveContainer" containerID="2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113" Jul 02 19:56:20.822109 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:20.822094 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113\": container with ID starting with 2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113 not found: ID does not exist" containerID="2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113" Jul 02 19:56:20.822144 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.822113 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113"} err="failed to get container status \"2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113\": rpc error: code = NotFound desc = could not find container \"2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113\": container with ID starting with 2f3cc77e0bd07609718be07d2fb19b930edf4e4d303ebc8888609cf7e1bc4113 not found: ID does not exist" Jul 02 19:56:20.822144 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.822127 2564 scope.go:117] "RemoveContainer" containerID="438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337" Jul 02 19:56:20.822339 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:20.822323 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337\": container with ID starting with 438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337 not found: ID does not exist" containerID="438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337" Jul 02 19:56:20.822386 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.822344 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337"} err="failed to get container status \"438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337\": rpc error: code = NotFound desc = could not find container \"438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337\": container with ID starting with 438c8af6fe208dcac0e94959d97840171acd09cba06b013cf6013b738ec9c337 not found: ID does not exist" Jul 02 19:56:20.822386 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.822373 2564 scope.go:117] "RemoveContainer" containerID="cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428" Jul 02 19:56:20.822654 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:20.822623 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428\": container with ID starting with cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428 not found: ID does not exist" containerID="cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428" Jul 02 19:56:20.822654 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.822655 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428"} err="failed to get container status \"cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428\": rpc error: code = NotFound desc = could not find container \"cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428\": container with ID starting with cbe2cbb55e521064dd1c67efeff033e85ddcd848f921f5a46ece33bd142c7428 not found: ID does not exist" Jul 02 19:56:20.822654 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.822682 2564 scope.go:117] "RemoveContainer" containerID="5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a" Jul 02 19:56:20.823123 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:20.822966 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a\": container with ID starting with 5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a not found: ID does not exist" containerID="5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a" Jul 02 19:56:20.823123 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:20.822994 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a"} err="failed to get container status \"5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a\": rpc error: code = NotFound desc = could not find container \"5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a\": container with ID starting with 5144b2b4cc289905d9674d78d827eaef21019e2fa892712e9bbe9c26e117f16a not found: ID does not exist" Jul 02 19:56:21.053099 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:21.053026 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" path="/var/lib/kubelet/pods/b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3/volumes" Jul 02 19:56:23.121813 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.121771 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:56:23.122299 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.122125 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="prometheus" containerID="cri-o://e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" gracePeriod=600 Jul 02 19:56:23.122535 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.122507 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-thanos" containerID="cri-o://73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" gracePeriod=600 Jul 02 19:56:23.122700 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.122666 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy" containerID="cri-o://b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" gracePeriod=600 Jul 02 19:56:23.122817 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.122795 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-web" containerID="cri-o://1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" gracePeriod=600 Jul 02 19:56:23.122907 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.122890 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="thanos-sidecar" containerID="cri-o://7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" gracePeriod=600 Jul 02 19:56:23.123014 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.122977 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="config-reloader" containerID="cri-o://03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" gracePeriod=600 Jul 02 19:56:23.378998 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.378933 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.527369 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527331 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-trusted-ca-bundle\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527391 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-metrics-client-ca\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527409 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-0\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527432 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-546zf\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-kube-api-access-546zf\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527492 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-kube-rbac-proxy\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527522 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-kubelet-serving-ca-bundle\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527546 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-grpc-tls\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527589 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527598 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-web-config\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527627 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-metrics-client-certs\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-2\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527723 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-config\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527749 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-config-out\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527777 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-serving-certs-ca-bundle\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527808 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527826 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-tls\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527850 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-thanos-prometheus-http-client-file\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-db\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527917 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-tls-assets\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527884 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:23.527937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.527937 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-1\") pod \"362f5208-8310-4e86-9736-2f4b25bd73da\" (UID: \"362f5208-8310-4e86-9736-2f4b25bd73da\") " Jul 02 19:56:23.528592 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.528177 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.528592 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.528194 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-metrics-client-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.528700 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.528606 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:23.528700 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.528481 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:23.528804 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.528715 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:23.529224 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.529086 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:23.529224 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.529176 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:23.530499 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.530450 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:56:23.530987 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.530954 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.531513 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.531488 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.531708 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.531605 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.531708 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.531684 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-config" (OuterVolumeSpecName: "config") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.533041 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.533006 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.533284 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.533239 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.533404 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.533367 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-kube-api-access-546zf" (OuterVolumeSpecName: "kube-api-access-546zf") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "kube-api-access-546zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:56:23.533736 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.533712 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.534713 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.534691 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.535069 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.535047 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-config-out" (OuterVolumeSpecName: "config-out") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:56:23.535069 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.535053 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:56:23.542477 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.542429 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-web-config" (OuterVolumeSpecName: "web-config") pod "362f5208-8310-4e86-9736-2f4b25bd73da" (UID: "362f5208-8310-4e86-9736-2f4b25bd73da"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:23.628970 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.628937 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-tls-assets\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.628970 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.628968 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.628970 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.628978 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.628989 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-546zf\" (UniqueName: \"kubernetes.io/projected/362f5208-8310-4e86-9736-2f4b25bd73da-kube-api-access-546zf\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.628998 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-kube-rbac-proxy\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629008 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629016 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-grpc-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629024 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629055 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-web-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629064 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629072 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-metrics-client-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629081 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629089 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629097 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-config-out\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629106 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/362f5208-8310-4e86-9736-2f4b25bd73da-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629117 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629127 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/362f5208-8310-4e86-9736-2f4b25bd73da-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.629212 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.629135 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/362f5208-8310-4e86-9736-2f4b25bd73da-prometheus-k8s-db\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 19:56:23.781259 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781222 2564 generic.go:356] "Generic (PLEG): container finished" podID="362f5208-8310-4e86-9736-2f4b25bd73da" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" exitCode=0 Jul 02 19:56:23.781259 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781255 2564 generic.go:356] "Generic (PLEG): container finished" podID="362f5208-8310-4e86-9736-2f4b25bd73da" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" exitCode=0 Jul 02 19:56:23.781259 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781263 2564 generic.go:356] "Generic (PLEG): container finished" podID="362f5208-8310-4e86-9736-2f4b25bd73da" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" exitCode=0 Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781272 2564 generic.go:356] "Generic (PLEG): container finished" podID="362f5208-8310-4e86-9736-2f4b25bd73da" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" exitCode=0 Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781280 2564 generic.go:356] "Generic (PLEG): container finished" podID="362f5208-8310-4e86-9736-2f4b25bd73da" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" exitCode=0 Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781288 2564 generic.go:356] "Generic (PLEG): container finished" podID="362f5208-8310-4e86-9736-2f4b25bd73da" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" exitCode=0 Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781318 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781488 2564 scope.go:117] "RemoveContainer" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781542 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} Jul 02 19:56:23.781586 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} Jul 02 19:56:23.782027 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} Jul 02 19:56:23.782027 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.781669 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"362f5208-8310-4e86-9736-2f4b25bd73da","Type":"ContainerDied","Data":"22df597d299fb10bde5647b9a66d00b22e6299d87c72f3c8938b2a230399273a"} Jul 02 19:56:23.789417 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.789395 2564 scope.go:117] "RemoveContainer" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.796113 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.796098 2564 scope.go:117] "RemoveContainer" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.802443 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.802426 2564 scope.go:117] "RemoveContainer" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.806246 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.806223 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:56:23.809707 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.809672 2564 scope.go:117] "RemoveContainer" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.812062 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.812042 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:56:23.816420 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.816404 2564 scope.go:117] "RemoveContainer" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.823220 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.823205 2564 scope.go:117] "RemoveContainer" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.829755 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.829740 2564 scope.go:117] "RemoveContainer" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.830002 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:23.829985 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": container with ID starting with 73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7 not found: ID does not exist" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.830064 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830008 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} err="failed to get container status \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": rpc error: code = NotFound desc = could not find container \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": container with ID starting with 73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7 not found: ID does not exist" Jul 02 19:56:23.830064 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830025 2564 scope.go:117] "RemoveContainer" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.830263 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:23.830245 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": container with ID starting with b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c not found: ID does not exist" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.830301 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830270 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} err="failed to get container status \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": rpc error: code = NotFound desc = could not find container \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": container with ID starting with b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c not found: ID does not exist" Jul 02 19:56:23.830301 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830289 2564 scope.go:117] "RemoveContainer" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.830552 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:23.830535 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": container with ID starting with 1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378 not found: ID does not exist" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.830622 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830560 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} err="failed to get container status \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": rpc error: code = NotFound desc = could not find container \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": container with ID starting with 1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378 not found: ID does not exist" Jul 02 19:56:23.830622 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830580 2564 scope.go:117] "RemoveContainer" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.830824 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:23.830808 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": container with ID starting with 7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56 not found: ID does not exist" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.830878 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830829 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} err="failed to get container status \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": rpc error: code = NotFound desc = could not find container \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": container with ID starting with 7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56 not found: ID does not exist" Jul 02 19:56:23.830878 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.830844 2564 scope.go:117] "RemoveContainer" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.831051 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:23.831034 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": container with ID starting with 03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1 not found: ID does not exist" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.831090 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831055 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} err="failed to get container status \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": rpc error: code = NotFound desc = could not find container \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": container with ID starting with 03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1 not found: ID does not exist" Jul 02 19:56:23.831090 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831069 2564 scope.go:117] "RemoveContainer" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.831269 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:23.831253 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": container with ID starting with e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c not found: ID does not exist" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.831328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831276 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} err="failed to get container status \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": rpc error: code = NotFound desc = could not find container \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": container with ID starting with e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c not found: ID does not exist" Jul 02 19:56:23.831328 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831295 2564 scope.go:117] "RemoveContainer" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.831645 ip-10-0-133-137 kubenswrapper[2564]: E0702 19:56:23.831629 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": container with ID starting with 0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472 not found: ID does not exist" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.831703 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831647 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472"} err="failed to get container status \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": rpc error: code = NotFound desc = could not find container \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": container with ID starting with 0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472 not found: ID does not exist" Jul 02 19:56:23.831703 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831660 2564 scope.go:117] "RemoveContainer" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.831879 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831863 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} err="failed to get container status \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": rpc error: code = NotFound desc = could not find container \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": container with ID starting with 73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7 not found: ID does not exist" Jul 02 19:56:23.831922 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.831879 2564 scope.go:117] "RemoveContainer" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.832120 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832103 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} err="failed to get container status \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": rpc error: code = NotFound desc = could not find container \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": container with ID starting with b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c not found: ID does not exist" Jul 02 19:56:23.832177 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832122 2564 scope.go:117] "RemoveContainer" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.832335 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832320 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} err="failed to get container status \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": rpc error: code = NotFound desc = could not find container \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": container with ID starting with 1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378 not found: ID does not exist" Jul 02 19:56:23.832380 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832335 2564 scope.go:117] "RemoveContainer" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.832576 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832554 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} err="failed to get container status \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": rpc error: code = NotFound desc = could not find container \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": container with ID starting with 7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56 not found: ID does not exist" Jul 02 19:56:23.832650 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832576 2564 scope.go:117] "RemoveContainer" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.832766 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832750 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} err="failed to get container status \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": rpc error: code = NotFound desc = could not find container \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": container with ID starting with 03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1 not found: ID does not exist" Jul 02 19:56:23.832810 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832766 2564 scope.go:117] "RemoveContainer" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.832970 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832950 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} err="failed to get container status \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": rpc error: code = NotFound desc = could not find container \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": container with ID starting with e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c not found: ID does not exist" Jul 02 19:56:23.833029 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.832971 2564 scope.go:117] "RemoveContainer" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.833165 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833150 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472"} err="failed to get container status \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": rpc error: code = NotFound desc = could not find container \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": container with ID starting with 0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472 not found: ID does not exist" Jul 02 19:56:23.833209 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833167 2564 scope.go:117] "RemoveContainer" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.833331 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833313 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} err="failed to get container status \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": rpc error: code = NotFound desc = could not find container \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": container with ID starting with 73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7 not found: ID does not exist" Jul 02 19:56:23.833369 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833332 2564 scope.go:117] "RemoveContainer" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.833539 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833517 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} err="failed to get container status \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": rpc error: code = NotFound desc = could not find container \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": container with ID starting with b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c not found: ID does not exist" Jul 02 19:56:23.833539 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833539 2564 scope.go:117] "RemoveContainer" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.833845 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833823 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} err="failed to get container status \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": rpc error: code = NotFound desc = could not find container \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": container with ID starting with 1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378 not found: ID does not exist" Jul 02 19:56:23.833919 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.833849 2564 scope.go:117] "RemoveContainer" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.834878 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.834433 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} err="failed to get container status \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": rpc error: code = NotFound desc = could not find container \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": container with ID starting with 7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56 not found: ID does not exist" Jul 02 19:56:23.834878 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.834482 2564 scope.go:117] "RemoveContainer" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.835057 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.834912 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} err="failed to get container status \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": rpc error: code = NotFound desc = could not find container \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": container with ID starting with 03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1 not found: ID does not exist" Jul 02 19:56:23.835057 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.834934 2564 scope.go:117] "RemoveContainer" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.835232 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835192 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} err="failed to get container status \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": rpc error: code = NotFound desc = could not find container \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": container with ID starting with e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c not found: ID does not exist" Jul 02 19:56:23.835232 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835231 2564 scope.go:117] "RemoveContainer" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.835479 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835442 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472"} err="failed to get container status \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": rpc error: code = NotFound desc = could not find container \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": container with ID starting with 0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472 not found: ID does not exist" Jul 02 19:56:23.835525 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835480 2564 scope.go:117] "RemoveContainer" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.835715 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835697 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} err="failed to get container status \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": rpc error: code = NotFound desc = could not find container \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": container with ID starting with 73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7 not found: ID does not exist" Jul 02 19:56:23.835715 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835715 2564 scope.go:117] "RemoveContainer" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.835927 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835907 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} err="failed to get container status \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": rpc error: code = NotFound desc = could not find container \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": container with ID starting with b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c not found: ID does not exist" Jul 02 19:56:23.835978 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835928 2564 scope.go:117] "RemoveContainer" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.835978 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.835969 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:56:23.836136 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836122 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} err="failed to get container status \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": rpc error: code = NotFound desc = could not find container \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": container with ID starting with 1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378 not found: ID does not exist" Jul 02 19:56:23.836177 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836137 2564 scope.go:117] "RemoveContainer" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.836240 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836223 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy" Jul 02 19:56:23.836240 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836242 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836256 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="config-reloader" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836264 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="config-reloader" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836278 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836286 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836294 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="prometheus" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836303 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="prometheus" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836316 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-web" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836324 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-web" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836334 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="init-config-reloader" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836345 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="init-config-reloader" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836345 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} err="failed to get container status \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": rpc error: code = NotFound desc = could not find container \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": container with ID starting with 7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56 not found: ID does not exist" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836354 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="alertmanager" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836362 2564 scope.go:117] "RemoveContainer" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.836406 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836363 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="alertmanager" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836414 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="thanos-sidecar" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836423 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="thanos-sidecar" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836431 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-web" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836436 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-web" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836444 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="config-reloader" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836451 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="config-reloader" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836479 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="prom-label-proxy" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836487 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="prom-label-proxy" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836506 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="init-config-reloader" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836515 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="init-config-reloader" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836524 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-metric" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836532 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-metric" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836542 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-thanos" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836550 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-thanos" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836624 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="thanos-sidecar" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836633 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-thanos" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836640 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836645 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="prom-label-proxy" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836651 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="prometheus" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836657 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy-web" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836653 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} err="failed to get container status \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": rpc error: code = NotFound desc = could not find container \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": container with ID starting with 03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1 not found: ID does not exist" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836663 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="alertmanager" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836669 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-metric" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836679 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="kube-rbac-proxy" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836686 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="config-reloader" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836691 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83bb63a-e34a-4a7a-9838-a3adc1c5f0d3" containerName="kube-rbac-proxy-web" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836696 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" containerName="config-reloader" Jul 02 19:56:23.836915 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836668 2564 scope.go:117] "RemoveContainer" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.836992 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} err="failed to get container status \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": rpc error: code = NotFound desc = could not find container \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": container with ID starting with e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c not found: ID does not exist" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.837015 2564 scope.go:117] "RemoveContainer" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.837441 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472"} err="failed to get container status \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": rpc error: code = NotFound desc = could not find container \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": container with ID starting with 0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472 not found: ID does not exist" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.837487 2564 scope.go:117] "RemoveContainer" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.837767 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} err="failed to get container status \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": rpc error: code = NotFound desc = could not find container \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": container with ID starting with 73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7 not found: ID does not exist" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.837793 2564 scope.go:117] "RemoveContainer" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838066 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} err="failed to get container status \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": rpc error: code = NotFound desc = could not find container \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": container with ID starting with b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c not found: ID does not exist" Jul 02 19:56:23.838101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838084 2564 scope.go:117] "RemoveContainer" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.838405 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838350 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} err="failed to get container status \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": rpc error: code = NotFound desc = could not find container \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": container with ID starting with 1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378 not found: ID does not exist" Jul 02 19:56:23.838405 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838371 2564 scope.go:117] "RemoveContainer" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.838607 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838589 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} err="failed to get container status \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": rpc error: code = NotFound desc = could not find container \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": container with ID starting with 7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56 not found: ID does not exist" Jul 02 19:56:23.838659 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838606 2564 scope.go:117] "RemoveContainer" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.838815 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838800 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} err="failed to get container status \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": rpc error: code = NotFound desc = could not find container \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": container with ID starting with 03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1 not found: ID does not exist" Jul 02 19:56:23.838855 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.838815 2564 scope.go:117] "RemoveContainer" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.839028 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839009 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} err="failed to get container status \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": rpc error: code = NotFound desc = could not find container \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": container with ID starting with e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c not found: ID does not exist" Jul 02 19:56:23.839078 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839028 2564 scope.go:117] "RemoveContainer" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.839239 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839224 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472"} err="failed to get container status \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": rpc error: code = NotFound desc = could not find container \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": container with ID starting with 0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472 not found: ID does not exist" Jul 02 19:56:23.839283 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839240 2564 scope.go:117] "RemoveContainer" containerID="73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7" Jul 02 19:56:23.839448 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839427 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7"} err="failed to get container status \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": rpc error: code = NotFound desc = could not find container \"73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7\": container with ID starting with 73dc122551d7caaeba175b020c579964bae6a7de5fd34d6b4cc79b92961c47f7 not found: ID does not exist" Jul 02 19:56:23.839517 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839449 2564 scope.go:117] "RemoveContainer" containerID="b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c" Jul 02 19:56:23.839701 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839684 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c"} err="failed to get container status \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": rpc error: code = NotFound desc = could not find container \"b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c\": container with ID starting with b45be7d5fa79812db976765cfb9ff8803a3018e2889596272eb71c1cb119d39c not found: ID does not exist" Jul 02 19:56:23.839763 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839701 2564 scope.go:117] "RemoveContainer" containerID="1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378" Jul 02 19:56:23.839900 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839883 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378"} err="failed to get container status \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": rpc error: code = NotFound desc = could not find container \"1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378\": container with ID starting with 1d552d1588e2aacb7fa131a8efbe349e726541aacf220319f77a7343fba53378 not found: ID does not exist" Jul 02 19:56:23.839937 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.839901 2564 scope.go:117] "RemoveContainer" containerID="7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56" Jul 02 19:56:23.840088 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.840072 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56"} err="failed to get container status \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": rpc error: code = NotFound desc = could not find container \"7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56\": container with ID starting with 7deacf78aa4eddf16ff0e6ec3dfa7aab25be3c5799cd6b6eb57af2e4e0824b56 not found: ID does not exist" Jul 02 19:56:23.840123 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.840088 2564 scope.go:117] "RemoveContainer" containerID="03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1" Jul 02 19:56:23.840278 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.840263 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1"} err="failed to get container status \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": rpc error: code = NotFound desc = could not find container \"03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1\": container with ID starting with 03e1494a04951d19f176b68d3a6af207126ff5c53fa986ea9a0cd1b176be4ab1 not found: ID does not exist" Jul 02 19:56:23.840278 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.840278 2564 scope.go:117] "RemoveContainer" containerID="e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c" Jul 02 19:56:23.840490 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.840474 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c"} err="failed to get container status \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": rpc error: code = NotFound desc = could not find container \"e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c\": container with ID starting with e9cb1cb9e4b808359ec0b9d5b8c78ab46d66cdb757ce455ae227da331e31266c not found: ID does not exist" Jul 02 19:56:23.840490 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.840490 2564 scope.go:117] "RemoveContainer" containerID="0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472" Jul 02 19:56:23.840708 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.840681 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472"} err="failed to get container status \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": rpc error: code = NotFound desc = could not find container \"0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472\": container with ID starting with 0832a4a3603687e3236d3d7c5ecdd45f4e6970c5d3d808af1a0d951a69844472 not found: ID does not exist" Jul 02 19:56:23.843210 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.843193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.846238 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.846220 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 02 19:56:23.846385 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.846253 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 02 19:56:23.847078 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847056 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 02 19:56:23.847166 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847113 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 02 19:56:23.847230 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847204 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 02 19:56:23.847647 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 02 19:56:23.847746 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847726 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d55rw\"" Jul 02 19:56:23.847865 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847849 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 02 19:56:23.847948 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847933 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 02 19:56:23.848003 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.847961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 02 19:56:23.848158 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.848142 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 02 19:56:23.848499 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.848442 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 02 19:56:23.848644 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.848598 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 02 19:56:23.848707 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.848602 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-vsbcmsnpq1dg\"" Jul 02 19:56:23.848707 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.848690 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 19:56:23.850520 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.850503 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 02 19:56:23.852368 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.852352 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 02 19:56:23.855721 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.855696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:56:23.931101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931278 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931278 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931278 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b40f16e3-ff32-4617-85f5-e507c2011906-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931278 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931278 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-web-config\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931438 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxz7\" (UniqueName: \"kubernetes.io/projected/b40f16e3-ff32-4617-85f5-e507c2011906-kube-api-access-vxxz7\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931675 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b40f16e3-ff32-4617-85f5-e507c2011906-config-out\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931675 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931675 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931675 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931675 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931675 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-config\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:23.931675 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:23.931611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032366 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032366 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-web-config\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxz7\" (UniqueName: \"kubernetes.io/projected/b40f16e3-ff32-4617-85f5-e507c2011906-kube-api-access-vxxz7\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b40f16e3-ff32-4617-85f5-e507c2011906-config-out\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.032619 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-config\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b40f16e3-ff32-4617-85f5-e507c2011906-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033080 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.032856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033451 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.033235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033451 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.033235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033451 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.033241 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033737 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.033711 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.033957 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.033933 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.035622 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.035589 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.035622 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.035605 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-web-config\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.035807 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.035763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.035807 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.035777 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.035932 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.035908 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.036547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.036041 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.036547 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.036223 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b40f16e3-ff32-4617-85f5-e507c2011906-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.036719 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.036575 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.036719 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.036639 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b40f16e3-ff32-4617-85f5-e507c2011906-config-out\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.038022 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.037991 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b40f16e3-ff32-4617-85f5-e507c2011906-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.038173 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.038142 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.038352 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.038332 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-config\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.038511 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.038495 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.038560 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.038497 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b40f16e3-ff32-4617-85f5-e507c2011906-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.041875 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.041852 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxz7\" (UniqueName: \"kubernetes.io/projected/b40f16e3-ff32-4617-85f5-e507c2011906-kube-api-access-vxxz7\") pod \"prometheus-k8s-0\" (UID: \"b40f16e3-ff32-4617-85f5-e507c2011906\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.153150 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.153122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:56:24.291720 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.291694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:56:24.294407 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:56:24.294380 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40f16e3_ff32_4617_85f5_e507c2011906.slice/crio-9ba87444e9048f2afaad1b3e9f9b824c260d3dfd9d07b1f7b85494ba63a06b36 WatchSource:0}: Error finding container 9ba87444e9048f2afaad1b3e9f9b824c260d3dfd9d07b1f7b85494ba63a06b36: Status 404 returned error can't find the container with id 9ba87444e9048f2afaad1b3e9f9b824c260d3dfd9d07b1f7b85494ba63a06b36 Jul 02 19:56:24.786618 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.786587 2564 generic.go:356] "Generic (PLEG): container finished" podID="b40f16e3-ff32-4617-85f5-e507c2011906" containerID="b63573448aaddaf9d908d115da500c69c2bf66ef8c2c5dba2dbbcfd5ccfa77a3" exitCode=0 Jul 02 19:56:24.786874 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.786628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerDied","Data":"b63573448aaddaf9d908d115da500c69c2bf66ef8c2c5dba2dbbcfd5ccfa77a3"} Jul 02 19:56:24.786874 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:24.786682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerStarted","Data":"9ba87444e9048f2afaad1b3e9f9b824c260d3dfd9d07b1f7b85494ba63a06b36"} Jul 02 19:56:25.055597 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.055565 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362f5208-8310-4e86-9736-2f4b25bd73da" path="/var/lib/kubelet/pods/362f5208-8310-4e86-9736-2f4b25bd73da/volumes" Jul 02 19:56:25.794207 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.794171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerStarted","Data":"7cafd45f70d301861c50e66ed06214cfb3eadfd7a3680f2e8ad4d8c983907618"} Jul 02 19:56:25.794599 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.794252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerStarted","Data":"623f8ecef27fa05f251736949ee0a713aa8a87d944a96137d910d2964f48362b"} Jul 02 19:56:25.794599 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.794290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerStarted","Data":"f49827b551491a90ef9f45aa33ab9ffd612c26489a8997a0140ce51b82b3074b"} Jul 02 19:56:25.794599 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.794329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerStarted","Data":"7953a36d98e304589113cdd5fd3ca83b73880b2bc914d56cecff735835930c39"} Jul 02 19:56:25.794599 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.794365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerStarted","Data":"1cb948346d8417728f7c9001283ae695b143c31c07b3a29fbfb35892fc214aa9"} Jul 02 19:56:25.794599 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.794399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b40f16e3-ff32-4617-85f5-e507c2011906","Type":"ContainerStarted","Data":"df64fdaf3c44ebe5bfb450fd459d5ef4a23b4df712e12119fabe22a5d3ac417a"} Jul 02 19:56:25.825169 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:25.825113 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8005552590000002 podStartE2EDuration="2.825092219s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.53696ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:56:25.82301033 +0000 UTC m=+165.191747066" watchObservedRunningTime="2026-07-02 19:56:25.825092219 +0000 UTC m=+165.193828945" Jul 02 19:56:29.153248 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:56:29.153208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:57:24.154028 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:57:24.153994 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:57:24.168980 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:57:24.168955 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:57:25.021135 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:57:25.021111 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:58:29.904605 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.904563 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j"] Jul 02 19:58:29.907933 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.907913 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:29.911109 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.911079 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 19:58:29.911557 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.911524 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 19:58:29.911676 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.911583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dmrc8\"" Jul 02 19:58:29.911676 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.911550 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 19:58:29.911972 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.911957 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 19:58:29.925531 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:29.925508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j"] Jul 02 19:58:30.059944 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.059914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-manifests\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.059944 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.059949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.060148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.060034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw752\" (UniqueName: \"kubernetes.io/projected/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-kube-api-access-pw752\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.060148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.060070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-webhook-cert\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.060148 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.060138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-tmp\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.161567 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.161479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-tmp\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.161739 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.161673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-manifests\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.161739 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.161723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.161885 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.161858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw752\" (UniqueName: \"kubernetes.io/projected/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-kube-api-access-pw752\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.161947 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.161906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-webhook-cert\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.162073 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.162050 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-manifests\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.163739 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.163714 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-tmp\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.164088 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.164067 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.164170 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.164156 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-webhook-cert\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.175379 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.175353 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw752\" (UniqueName: \"kubernetes.io/projected/a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b-kube-api-access-pw752\") pod \"opendatahub-operator-controller-manager-844d4f46fb-7zt4j\" (UID: \"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b\") " pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.219822 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.219795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:30.356296 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.356273 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j"] Jul 02 19:58:30.358875 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:58:30.358850 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f30d1e_c8bc_4ee3_8d78_9e9bf8e1e80b.slice/crio-0d1a8b103f3070b18c4ca3abaaf788b80b3f2265621af15b5f198a7c1e7ca2dc WatchSource:0}: Error finding container 0d1a8b103f3070b18c4ca3abaaf788b80b3f2265621af15b5f198a7c1e7ca2dc: Status 404 returned error can't find the container with id 0d1a8b103f3070b18c4ca3abaaf788b80b3f2265621af15b5f198a7c1e7ca2dc Jul 02 19:58:30.639432 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.639407 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt"] Jul 02 19:58:30.643648 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.643631 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.646660 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.646628 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tq2kn\"" Jul 02 19:58:30.646816 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.646799 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 19:58:30.646901 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.646873 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 19:58:30.646901 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.646889 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:58:30.647005 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.646889 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 19:58:30.647005 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.646897 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 19:58:30.657174 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.657154 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt"] Jul 02 19:58:30.766154 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.766120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcq2\" (UniqueName: \"kubernetes.io/projected/78b5a514-d28c-48a6-a348-c380b9c16a03-kube-api-access-6bcq2\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.766315 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.766164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b5a514-d28c-48a6-a348-c380b9c16a03-cert\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.766315 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.766203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78b5a514-d28c-48a6-a348-c380b9c16a03-metrics-cert\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.766315 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.766252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78b5a514-d28c-48a6-a348-c380b9c16a03-manager-config\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.867703 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.867649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcq2\" (UniqueName: \"kubernetes.io/projected/78b5a514-d28c-48a6-a348-c380b9c16a03-kube-api-access-6bcq2\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.867908 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.867782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b5a514-d28c-48a6-a348-c380b9c16a03-cert\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.867908 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.867874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78b5a514-d28c-48a6-a348-c380b9c16a03-metrics-cert\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.868057 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.867909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78b5a514-d28c-48a6-a348-c380b9c16a03-manager-config\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.869008 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.868982 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78b5a514-d28c-48a6-a348-c380b9c16a03-manager-config\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.870788 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.870762 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78b5a514-d28c-48a6-a348-c380b9c16a03-metrics-cert\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.870919 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.870853 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b5a514-d28c-48a6-a348-c380b9c16a03-cert\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.879279 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.879255 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcq2\" (UniqueName: \"kubernetes.io/projected/78b5a514-d28c-48a6-a348-c380b9c16a03-kube-api-access-6bcq2\") pod \"lws-controller-manager-56c987df7b-7mmbt\" (UID: \"78b5a514-d28c-48a6-a348-c380b9c16a03\") " pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:30.953575 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:30.953480 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:31.120983 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:31.120958 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt"] Jul 02 19:58:31.123244 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:58:31.123208 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b5a514_d28c_48a6_a348_c380b9c16a03.slice/crio-d012dd031610f91403749fb35818a450d658cd8c63f2c0769bcc2af8aecb0830 WatchSource:0}: Error finding container d012dd031610f91403749fb35818a450d658cd8c63f2c0769bcc2af8aecb0830: Status 404 returned error can't find the container with id d012dd031610f91403749fb35818a450d658cd8c63f2c0769bcc2af8aecb0830 Jul 02 19:58:31.244318 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:31.244191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" event={"ID":"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b","Type":"ContainerStarted","Data":"0d1a8b103f3070b18c4ca3abaaf788b80b3f2265621af15b5f198a7c1e7ca2dc"} Jul 02 19:58:31.245771 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:31.245737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" event={"ID":"78b5a514-d28c-48a6-a348-c380b9c16a03","Type":"ContainerStarted","Data":"d012dd031610f91403749fb35818a450d658cd8c63f2c0769bcc2af8aecb0830"} Jul 02 19:58:34.258859 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:34.258805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" event={"ID":"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b","Type":"ContainerStarted","Data":"40764e84a8fc4cb597e0735735bd2f67232ef15b82d2b1e36bfe144ac9d8aecf"} Jul 02 19:58:35.264264 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:35.264231 2564 generic.go:356] "Generic (PLEG): container finished" podID="a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b" containerID="40764e84a8fc4cb597e0735735bd2f67232ef15b82d2b1e36bfe144ac9d8aecf" exitCode=0 Jul 02 19:58:35.264786 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:35.264317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" event={"ID":"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b","Type":"ContainerDied","Data":"40764e84a8fc4cb597e0735735bd2f67232ef15b82d2b1e36bfe144ac9d8aecf"} Jul 02 19:58:35.265932 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:35.265911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" event={"ID":"78b5a514-d28c-48a6-a348-c380b9c16a03","Type":"ContainerStarted","Data":"e16a1941cd3f160e5f0ac04f702a92ae1af0dd66ff004dceed290dd6ad16f17c"} Jul 02 19:58:35.266110 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:35.266039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:35.309765 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:35.309650 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" podStartSLOduration=1.339910153 podStartE2EDuration="5.30963378s" totalImagesPullingTime="3.969723627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:58:35.308640221 +0000 UTC m=+294.677376948" watchObservedRunningTime="2026-07-02 19:58:35.30963378 +0000 UTC m=+294.678370505" Jul 02 19:58:36.272340 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:36.272303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" event={"ID":"a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b","Type":"ContainerStarted","Data":"c99c4e4b208f941059496ca0842f7b1f98af6faac3ebf9cf6f12f4fb1f771050"} Jul 02 19:58:36.272824 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:36.272476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:36.294155 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:36.294106 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" podStartSLOduration=2.582277163 podStartE2EDuration="7.294087462s" totalImagesPullingTime="3.345366822s" totalInitContainerRuntime="1.366443477s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:58:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:58:36.292824906 +0000 UTC m=+295.661561630" watchObservedRunningTime="2026-07-02 19:58:36.294087462 +0000 UTC m=+295.662824188" Jul 02 19:58:40.950905 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:40.950877 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 19:58:46.274801 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:46.274769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56c987df7b-7mmbt" Jul 02 19:58:47.278873 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:47.278842 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-844d4f46fb-7zt4j" Jul 02 19:58:49.815452 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.815410 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw"] Jul 02 19:58:49.820343 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.820321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:49.827571 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.827551 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 19:58:49.839364 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.839343 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 19:58:49.839647 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.839490 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 19:58:49.840501 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.840485 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jn846\"" Jul 02 19:58:49.840962 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.840941 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw"] Jul 02 19:58:49.841244 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.841230 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 19:58:49.937164 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.937134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwczr\" (UniqueName: \"kubernetes.io/projected/cd296c3b-667a-4360-9d8e-6df95d158041-kube-api-access-nwczr\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:49.937345 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.937172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd296c3b-667a-4360-9d8e-6df95d158041-tmp\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:49.937345 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:49.937291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd296c3b-667a-4360-9d8e-6df95d158041-tls-certs\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.038628 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.038594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwczr\" (UniqueName: \"kubernetes.io/projected/cd296c3b-667a-4360-9d8e-6df95d158041-kube-api-access-nwczr\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.038792 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.038637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd296c3b-667a-4360-9d8e-6df95d158041-tmp\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.038792 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.038717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd296c3b-667a-4360-9d8e-6df95d158041-tls-certs\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.040900 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.040870 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd296c3b-667a-4360-9d8e-6df95d158041-tmp\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.041101 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.041085 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd296c3b-667a-4360-9d8e-6df95d158041-tls-certs\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.052008 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.051984 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwczr\" (UniqueName: \"kubernetes.io/projected/cd296c3b-667a-4360-9d8e-6df95d158041-kube-api-access-nwczr\") pod \"kube-auth-proxy-67b994b5d7-n85jw\" (UID: \"cd296c3b-667a-4360-9d8e-6df95d158041\") " pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.130111 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.130024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" Jul 02 19:58:50.261332 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.261306 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw"] Jul 02 19:58:50.263456 ip-10-0-133-137 kubenswrapper[2564]: W0702 19:58:50.263417 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd296c3b_667a_4360_9d8e_6df95d158041.slice/crio-20003a8089e029847d9e80498c581a8cfca08f2b9589fcc156e2fb85905d9da1 WatchSource:0}: Error finding container 20003a8089e029847d9e80498c581a8cfca08f2b9589fcc156e2fb85905d9da1: Status 404 returned error can't find the container with id 20003a8089e029847d9e80498c581a8cfca08f2b9589fcc156e2fb85905d9da1 Jul 02 19:58:50.265214 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.265195 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:58:50.327668 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:50.327632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" event={"ID":"cd296c3b-667a-4360-9d8e-6df95d158041","Type":"ContainerStarted","Data":"20003a8089e029847d9e80498c581a8cfca08f2b9589fcc156e2fb85905d9da1"} Jul 02 19:58:53.340867 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:53.340826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" event={"ID":"cd296c3b-667a-4360-9d8e-6df95d158041","Type":"ContainerStarted","Data":"f05fdd020bb2324999068cc4cc60f953f82f2e7b6bea0272ae413f1f72874241"} Jul 02 19:58:53.358292 ip-10-0-133-137 kubenswrapper[2564]: I0702 19:58:53.358229 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67b994b5d7-n85jw" podStartSLOduration=1.830987061 podStartE2EDuration="4.358212506s" totalImagesPullingTime="2.527225445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:58:53.357264757 +0000 UTC m=+312.726001482" watchObservedRunningTime="2026-07-02 19:58:53.358212506 +0000 UTC m=+312.726949231" Jul 02 20:00:29.074440 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.074405 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7"] Jul 02 20:00:29.077694 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.077670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" Jul 02 20:00:29.080671 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.080655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 20:00:29.080867 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.080853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-k78g4\"" Jul 02 20:00:29.080951 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.080873 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 02 20:00:29.080951 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.080886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 20:00:29.088101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.088078 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7"] Jul 02 20:00:29.187801 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.187770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclb6\" (UniqueName: \"kubernetes.io/projected/2a092568-d472-4840-96ca-b3d179b5cded-kube-api-access-sclb6\") pod \"dns-operator-controller-manager-864dd78c4c-blhh7\" (UID: \"2a092568-d472-4840-96ca-b3d179b5cded\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" Jul 02 20:00:29.288595 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.288556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclb6\" (UniqueName: \"kubernetes.io/projected/2a092568-d472-4840-96ca-b3d179b5cded-kube-api-access-sclb6\") pod \"dns-operator-controller-manager-864dd78c4c-blhh7\" (UID: \"2a092568-d472-4840-96ca-b3d179b5cded\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" Jul 02 20:00:29.298757 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.298733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclb6\" (UniqueName: \"kubernetes.io/projected/2a092568-d472-4840-96ca-b3d179b5cded-kube-api-access-sclb6\") pod \"dns-operator-controller-manager-864dd78c4c-blhh7\" (UID: \"2a092568-d472-4840-96ca-b3d179b5cded\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" Jul 02 20:00:29.388024 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.387940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" Jul 02 20:00:29.512407 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.512361 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7"] Jul 02 20:00:29.517906 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:00:29.517877 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a092568_d472_4840_96ca_b3d179b5cded.slice/crio-687c88464f2658f8166beb86e7211dd12818a6e03ff99e5b005ce6e1518ee8fe WatchSource:0}: Error finding container 687c88464f2658f8166beb86e7211dd12818a6e03ff99e5b005ce6e1518ee8fe: Status 404 returned error can't find the container with id 687c88464f2658f8166beb86e7211dd12818a6e03ff99e5b005ce6e1518ee8fe Jul 02 20:00:29.723507 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:29.723456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" event={"ID":"2a092568-d472-4840-96ca-b3d179b5cded","Type":"ContainerStarted","Data":"687c88464f2658f8166beb86e7211dd12818a6e03ff99e5b005ce6e1518ee8fe"} Jul 02 20:00:32.736399 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:32.736362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" event={"ID":"2a092568-d472-4840-96ca-b3d179b5cded","Type":"ContainerStarted","Data":"773987812d9c663c196b8cc5118d1a378e11239a9a3b2f332ec89c2e98d93f50"} Jul 02 20:00:32.736787 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:32.736439 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" Jul 02 20:00:32.780435 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:32.780385 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" podStartSLOduration=1.033239097 podStartE2EDuration="3.780371974s" totalImagesPullingTime="2.747132877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:00:32.778885392 +0000 UTC m=+412.147622117" watchObservedRunningTime="2026-07-02 20:00:32.780371974 +0000 UTC m=+412.149108700" Jul 02 20:00:35.373880 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.373847 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq"] Jul 02 20:00:35.377160 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.377143 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.380306 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.380285 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 02 20:00:35.380549 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.380527 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 02 20:00:35.380643 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.380529 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7dvzc\"" Jul 02 20:00:35.387925 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.387904 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq"] Jul 02 20:00:35.437271 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.437239 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/07bf8241-38ab-4e69-81f4-6b9a9a872762-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.437453 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.437289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07bf8241-38ab-4e69-81f4-6b9a9a872762-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.437453 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.437318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/07bf8241-38ab-4e69-81f4-6b9a9a872762-kube-api-access-66c2c\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.538787 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.538736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/07bf8241-38ab-4e69-81f4-6b9a9a872762-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.538787 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.538799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07bf8241-38ab-4e69-81f4-6b9a9a872762-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.538992 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.538829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/07bf8241-38ab-4e69-81f4-6b9a9a872762-kube-api-access-66c2c\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.539476 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.539443 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07bf8241-38ab-4e69-81f4-6b9a9a872762-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.541092 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.541072 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/07bf8241-38ab-4e69-81f4-6b9a9a872762-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.548833 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.548808 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66c2c\" (UniqueName: \"kubernetes.io/projected/07bf8241-38ab-4e69-81f4-6b9a9a872762-kube-api-access-66c2c\") pod \"kuadrant-console-plugin-66b5774985-h5qxq\" (UID: \"07bf8241-38ab-4e69-81f4-6b9a9a872762\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.686537 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.686504 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" Jul 02 20:00:35.815432 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:35.815409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq"] Jul 02 20:00:35.817491 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:00:35.817452 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07bf8241_38ab_4e69_81f4_6b9a9a872762.slice/crio-52c44b886834bead1ca80b3c63f812dabb80260fbaa697dba88fb99e533881b4 WatchSource:0}: Error finding container 52c44b886834bead1ca80b3c63f812dabb80260fbaa697dba88fb99e533881b4: Status 404 returned error can't find the container with id 52c44b886834bead1ca80b3c63f812dabb80260fbaa697dba88fb99e533881b4 Jul 02 20:00:36.757717 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:36.757664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" event={"ID":"07bf8241-38ab-4e69-81f4-6b9a9a872762","Type":"ContainerStarted","Data":"52c44b886834bead1ca80b3c63f812dabb80260fbaa697dba88fb99e533881b4"} Jul 02 20:00:43.743621 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:43.743589 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-blhh7" Jul 02 20:00:46.046922 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.046882 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p"] Jul 02 20:00:46.058014 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.057749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:46.061330 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.061174 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nzpwx\"" Jul 02 20:00:46.069087 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.069059 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p"] Jul 02 20:00:46.137184 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.137148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69b5876e-17c5-472e-8a73-f34cfc57300c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gng6p\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:46.137360 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.137324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb2tk\" (UniqueName: \"kubernetes.io/projected/69b5876e-17c5-472e-8a73-f34cfc57300c-kube-api-access-vb2tk\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gng6p\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:46.238726 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.238680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2tk\" (UniqueName: \"kubernetes.io/projected/69b5876e-17c5-472e-8a73-f34cfc57300c-kube-api-access-vb2tk\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gng6p\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:46.238904 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.238844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69b5876e-17c5-472e-8a73-f34cfc57300c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gng6p\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:46.239255 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.239232 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69b5876e-17c5-472e-8a73-f34cfc57300c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gng6p\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:46.250652 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.250623 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2tk\" (UniqueName: \"kubernetes.io/projected/69b5876e-17c5-472e-8a73-f34cfc57300c-kube-api-access-vb2tk\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gng6p\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:46.371221 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:46.371138 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:00:59.731059 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:59.731032 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p"] Jul 02 20:00:59.733886 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:00:59.733852 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b5876e_17c5_472e_8a73_f34cfc57300c.slice/crio-6d6bc4040d19a3e0f6c55496df22a6db2c835670274aa3f1c221ada12a8e3002 WatchSource:0}: Error finding container 6d6bc4040d19a3e0f6c55496df22a6db2c835670274aa3f1c221ada12a8e3002: Status 404 returned error can't find the container with id 6d6bc4040d19a3e0f6c55496df22a6db2c835670274aa3f1c221ada12a8e3002 Jul 02 20:00:59.866934 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:00:59.866857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" event={"ID":"69b5876e-17c5-472e-8a73-f34cfc57300c","Type":"ContainerStarted","Data":"6d6bc4040d19a3e0f6c55496df22a6db2c835670274aa3f1c221ada12a8e3002"} Jul 02 20:01:00.872455 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:00.872418 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" event={"ID":"07bf8241-38ab-4e69-81f4-6b9a9a872762","Type":"ContainerStarted","Data":"04a7089a81a35ba1f8b54322dd64bf988e5d778984ad3d6c33df2e9ef3aac111"} Jul 02 20:01:00.888140 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:00.888079 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h5qxq" podStartSLOduration=1.7536225920000001 podStartE2EDuration="25.888062149s" totalImagesPullingTime="24.134439557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:01:00.887567067 +0000 UTC m=+440.256303797" watchObservedRunningTime="2026-07-02 20:01:00.888062149 +0000 UTC m=+440.256798873" Jul 02 20:01:05.898604 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:05.898567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" event={"ID":"69b5876e-17c5-472e-8a73-f34cfc57300c","Type":"ContainerStarted","Data":"acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e"} Jul 02 20:01:05.899108 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:05.898692 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:01:05.923277 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:05.923230 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" podStartSLOduration=14.262080609 podStartE2EDuration="19.923213468s" totalImagesPullingTime="5.661132859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:01:05.921942718 +0000 UTC m=+445.290679443" watchObservedRunningTime="2026-07-02 20:01:05.923213468 +0000 UTC m=+445.291950192" Jul 02 20:01:16.905674 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:16.905641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:01:28.905420 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:28.905383 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p"] Jul 02 20:01:28.905896 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:28.905675 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" podUID="69b5876e-17c5-472e-8a73-f34cfc57300c" containerName="manager" containerID="cri-o://acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e" gracePeriod=10 Jul 02 20:01:29.158084 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:29.158011 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:01:29.231359 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:29.231290 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69b5876e-17c5-472e-8a73-f34cfc57300c-extensions-socket-volume\") pod \"69b5876e-17c5-472e-8a73-f34cfc57300c\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " Jul 02 20:01:29.231531 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:29.231430 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb2tk\" (UniqueName: \"kubernetes.io/projected/69b5876e-17c5-472e-8a73-f34cfc57300c-kube-api-access-vb2tk\") pod \"69b5876e-17c5-472e-8a73-f34cfc57300c\" (UID: \"69b5876e-17c5-472e-8a73-f34cfc57300c\") " Jul 02 20:01:29.231791 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:29.231765 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b5876e-17c5-472e-8a73-f34cfc57300c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "69b5876e-17c5-472e-8a73-f34cfc57300c" (UID: "69b5876e-17c5-472e-8a73-f34cfc57300c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:01:29.233635 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:29.233608 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b5876e-17c5-472e-8a73-f34cfc57300c-kube-api-access-vb2tk" (OuterVolumeSpecName: "kube-api-access-vb2tk") pod "69b5876e-17c5-472e-8a73-f34cfc57300c" (UID: "69b5876e-17c5-472e-8a73-f34cfc57300c"). InnerVolumeSpecName "kube-api-access-vb2tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:01:29.332709 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:29.332671 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb2tk\" (UniqueName: \"kubernetes.io/projected/69b5876e-17c5-472e-8a73-f34cfc57300c-kube-api-access-vb2tk\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:01:29.332709 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:29.332705 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69b5876e-17c5-472e-8a73-f34cfc57300c-extensions-socket-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:01:30.011492 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.011432 2564 generic.go:356] "Generic (PLEG): container finished" podID="69b5876e-17c5-472e-8a73-f34cfc57300c" containerID="acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e" exitCode=0 Jul 02 20:01:30.011897 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.011539 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" Jul 02 20:01:30.011897 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.011561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" event={"ID":"69b5876e-17c5-472e-8a73-f34cfc57300c","Type":"ContainerDied","Data":"acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e"} Jul 02 20:01:30.011897 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.011649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p" event={"ID":"69b5876e-17c5-472e-8a73-f34cfc57300c","Type":"ContainerDied","Data":"6d6bc4040d19a3e0f6c55496df22a6db2c835670274aa3f1c221ada12a8e3002"} Jul 02 20:01:30.011897 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.011675 2564 scope.go:117] "RemoveContainer" containerID="acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e" Jul 02 20:01:30.021168 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.021146 2564 scope.go:117] "RemoveContainer" containerID="acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e" Jul 02 20:01:30.021478 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:01:30.021441 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e\": container with ID starting with acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e not found: ID does not exist" containerID="acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e" Jul 02 20:01:30.021577 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.021494 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e"} err="failed to get container status \"acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e\": rpc error: code = NotFound desc = could not find container \"acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e\": container with ID starting with acd7850c6d3b7406c689aeebc646502ccfeff9eb05edadeca56c0da781b13e5e not found: ID does not exist" Jul 02 20:01:30.034744 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.034713 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p"] Jul 02 20:01:30.040726 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:30.040704 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gng6p"] Jul 02 20:01:31.055496 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:31.055436 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b5876e-17c5-472e-8a73-f34cfc57300c" path="/var/lib/kubelet/pods/69b5876e-17c5-472e-8a73-f34cfc57300c/volumes" Jul 02 20:01:37.569006 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.568973 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:01:37.569482 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.569356 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b5876e-17c5-472e-8a73-f34cfc57300c" containerName="manager" Jul 02 20:01:37.569482 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.569368 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b5876e-17c5-472e-8a73-f34cfc57300c" containerName="manager" Jul 02 20:01:37.569482 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.569422 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b5876e-17c5-472e-8a73-f34cfc57300c" containerName="manager" Jul 02 20:01:37.572573 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.572553 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:37.575534 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.575512 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 20:01:37.592654 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.592623 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:01:37.655242 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.655204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:01:37.707401 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.707373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52e0e3be-3979-45bf-8700-b800a10576a0-config-file\") pod \"limitador-limitador-54d6d988d5-jq7cl\" (UID: \"52e0e3be-3979-45bf-8700-b800a10576a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:37.707561 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.707417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr4zw\" (UniqueName: \"kubernetes.io/projected/52e0e3be-3979-45bf-8700-b800a10576a0-kube-api-access-dr4zw\") pod \"limitador-limitador-54d6d988d5-jq7cl\" (UID: \"52e0e3be-3979-45bf-8700-b800a10576a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:37.808396 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.808354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52e0e3be-3979-45bf-8700-b800a10576a0-config-file\") pod \"limitador-limitador-54d6d988d5-jq7cl\" (UID: \"52e0e3be-3979-45bf-8700-b800a10576a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:37.808396 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.808400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr4zw\" (UniqueName: \"kubernetes.io/projected/52e0e3be-3979-45bf-8700-b800a10576a0-kube-api-access-dr4zw\") pod \"limitador-limitador-54d6d988d5-jq7cl\" (UID: \"52e0e3be-3979-45bf-8700-b800a10576a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:37.809106 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.809084 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52e0e3be-3979-45bf-8700-b800a10576a0-config-file\") pod \"limitador-limitador-54d6d988d5-jq7cl\" (UID: \"52e0e3be-3979-45bf-8700-b800a10576a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:37.816983 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.816958 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr4zw\" (UniqueName: \"kubernetes.io/projected/52e0e3be-3979-45bf-8700-b800a10576a0-kube-api-access-dr4zw\") pod \"limitador-limitador-54d6d988d5-jq7cl\" (UID: \"52e0e3be-3979-45bf-8700-b800a10576a0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:37.882693 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:37.882596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:38.013566 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:38.013505 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:01:38.016360 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:01:38.016321 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e0e3be_3979_45bf_8700_b800a10576a0.slice/crio-14ab264ce169f24525bb87745e7d3b58c2f83dd17f52c4069c636d6452477522 WatchSource:0}: Error finding container 14ab264ce169f24525bb87745e7d3b58c2f83dd17f52c4069c636d6452477522: Status 404 returned error can't find the container with id 14ab264ce169f24525bb87745e7d3b58c2f83dd17f52c4069c636d6452477522 Jul 02 20:01:38.046935 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:38.046882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" event={"ID":"52e0e3be-3979-45bf-8700-b800a10576a0","Type":"ContainerStarted","Data":"14ab264ce169f24525bb87745e7d3b58c2f83dd17f52c4069c636d6452477522"} Jul 02 20:01:41.064547 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:41.064513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" event={"ID":"52e0e3be-3979-45bf-8700-b800a10576a0","Type":"ContainerStarted","Data":"871a9747a164f0621d0e51928f5b76ea65cc020544e3a3da144faef3bd87f83b"} Jul 02 20:01:41.065007 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:41.064650 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:01:41.081048 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:41.081004 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" podStartSLOduration=1.578389306 podStartE2EDuration="4.080991154s" totalImagesPullingTime="2.502601848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:01:41.079741278 +0000 UTC m=+480.448478003" watchObservedRunningTime="2026-07-02 20:01:41.080991154 +0000 UTC m=+480.449727880" Jul 02 20:01:52.068842 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:01:52.068810 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jq7cl" Jul 02 20:02:51.280738 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:02:51.280656 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:03:17.402162 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:03:17.402125 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:03:42.207861 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:03:42.207820 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:03:50.811290 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:03:50.811252 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:04:06.076971 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.076908 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq"] Jul 02 20:04:06.080402 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.080378 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.083117 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.083094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 20:04:06.083117 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.083108 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 20:04:06.084435 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.084413 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-whwbw\"" Jul 02 20:04:06.084566 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.084412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 02 20:04:06.090895 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.090873 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq"] Jul 02 20:04:06.091261 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.091240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.091338 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.091279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.091338 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.091322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.091444 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.091365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.091444 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.091428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.091571 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.091518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s5hm\" (UniqueName: \"kubernetes.io/projected/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-kube-api-access-9s5hm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.191898 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.191857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s5hm\" (UniqueName: \"kubernetes.io/projected/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-kube-api-access-9s5hm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.191898 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.191902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.192162 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.191928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.192162 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.191947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.192162 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.191981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.192162 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.192008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.192374 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.192351 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.192426 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.192409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.192556 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.192534 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.194345 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.194323 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.194500 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.194483 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.200305 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.200277 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s5hm\" (UniqueName: \"kubernetes.io/projected/f9c2a195-f8fa-409d-ac08-5a47ff614b9a-kube-api-access-9s5hm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq\" (UID: \"f9c2a195-f8fa-409d-ac08-5a47ff614b9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.394206 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.394180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:06.519774 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.519737 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq"] Jul 02 20:04:06.522707 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:04:06.522675 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c2a195_f8fa_409d_ac08_5a47ff614b9a.slice/crio-c94340bf408c710a62b11d3c05055d3add1033b5db4ad096c78b3ebc37a5c46f WatchSource:0}: Error finding container c94340bf408c710a62b11d3c05055d3add1033b5db4ad096c78b3ebc37a5c46f: Status 404 returned error can't find the container with id c94340bf408c710a62b11d3c05055d3add1033b5db4ad096c78b3ebc37a5c46f Jul 02 20:04:06.523982 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.523966 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:04:06.655964 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:06.655885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" event={"ID":"f9c2a195-f8fa-409d-ac08-5a47ff614b9a","Type":"ContainerStarted","Data":"c94340bf408c710a62b11d3c05055d3add1033b5db4ad096c78b3ebc37a5c46f"} Jul 02 20:04:07.615032 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:07.615000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:04:07.661430 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:07.661396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" event={"ID":"f9c2a195-f8fa-409d-ac08-5a47ff614b9a","Type":"ContainerStarted","Data":"48a0f50304086fd16f987f9012875e5eb841e86137d75e8063f84e3fc72334c4"} Jul 02 20:04:07.679583 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:07.679411 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" podStartSLOduration=0.608477754 podStartE2EDuration="1.679390808s" totalImagesPullingTime="1.070913054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:04:07.679051951 +0000 UTC m=+627.047788676" watchObservedRunningTime="2026-07-02 20:04:07.679390808 +0000 UTC m=+627.048127530" Jul 02 20:04:16.049398 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:16.049363 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:04:16.394723 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:16.394692 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:16.394880 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:16.394763 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:16.407148 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:16.407122 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:16.712001 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:16.711914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq" Jul 02 20:04:25.877311 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.877275 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll"] Jul 02 20:04:25.880695 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.880676 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:25.883835 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.883810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 02 20:04:25.891167 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.891144 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll"] Jul 02 20:04:25.965296 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.965263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:25.965296 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.965299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f432d0-a6da-4689-a236-db9d481d29e8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:25.965542 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.965327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6dt\" (UniqueName: \"kubernetes.io/projected/48f432d0-a6da-4689-a236-db9d481d29e8-kube-api-access-xz6dt\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:25.965542 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.965404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:25.965542 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.965438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:25.965542 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:25.965453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066210 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066400 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066400 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f432d0-a6da-4689-a236-db9d481d29e8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066400 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6dt\" (UniqueName: \"kubernetes.io/projected/48f432d0-a6da-4689-a236-db9d481d29e8-kube-api-access-xz6dt\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066400 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066677 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066772 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066747 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066993 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066927 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.066993 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.066927 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.068681 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.068655 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f432d0-a6da-4689-a236-db9d481d29e8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.069419 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.069385 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f432d0-a6da-4689-a236-db9d481d29e8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.074888 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.074868 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6dt\" (UniqueName: \"kubernetes.io/projected/48f432d0-a6da-4689-a236-db9d481d29e8-kube-api-access-xz6dt\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll\" (UID: \"48f432d0-a6da-4689-a236-db9d481d29e8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.190399 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.190365 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:26.318953 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.318932 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll"] Jul 02 20:04:26.321252 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:04:26.321225 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48f432d0_a6da_4689_a236_db9d481d29e8.slice/crio-40b92ea39f3e5b7f789c70ca8b4ca3841db69730ca3b926f875eb5e90a9ff08c WatchSource:0}: Error finding container 40b92ea39f3e5b7f789c70ca8b4ca3841db69730ca3b926f875eb5e90a9ff08c: Status 404 returned error can't find the container with id 40b92ea39f3e5b7f789c70ca8b4ca3841db69730ca3b926f875eb5e90a9ff08c Jul 02 20:04:26.744307 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.744266 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" event={"ID":"48f432d0-a6da-4689-a236-db9d481d29e8","Type":"ContainerStarted","Data":"a6cf80d5f1fed57ad11b4831a0122c9fe7585c1cbc66034cb2d5b110b1c6626e"} Jul 02 20:04:26.744494 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.744315 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" event={"ID":"48f432d0-a6da-4689-a236-db9d481d29e8","Type":"ContainerStarted","Data":"40b92ea39f3e5b7f789c70ca8b4ca3841db69730ca3b926f875eb5e90a9ff08c"} Jul 02 20:04:26.763069 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:26.763021 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" podStartSLOduration=1.560405026 podStartE2EDuration="1.763007296s" totalImagesPullingTime="202.60227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:04:26.760734114 +0000 UTC m=+646.129470840" watchObservedRunningTime="2026-07-02 20:04:26.763007296 +0000 UTC m=+646.131744021" Jul 02 20:04:28.412939 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:28.412906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:04:36.190663 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:36.190618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:36.191063 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:36.190690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:36.202960 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:36.202937 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:04:36.801615 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:04:36.801588 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll" Jul 02 20:06:03.409578 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:06:03.409492 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:06:14.110494 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:06:14.110442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:06:25.008654 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:06:25.008623 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:06:34.303668 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:06:34.303636 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:06:44.500931 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:06:44.500898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:07:51.415274 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:07:51.415239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:08:00.006058 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:08:00.006018 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:08:09.112510 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:08:09.112477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:08:14.513407 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:08:14.513370 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:08:25.219252 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:08:25.219218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:08:34.018299 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:08:34.018223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:08:44.722374 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:08:44.722340 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:08:54.512414 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:08:54.512382 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:09:16.520773 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:09:16.520738 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:09:28.015383 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:09:28.015345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:09:51.413554 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:09:51.413507 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:10:01.909792 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:10:01.909754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:10:28.021687 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:10:28.021637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:10:38.923743 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:10:38.923708 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:10:47.816957 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:10:47.816924 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:10:56.522809 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:10:56.522776 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:12:44.712652 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:12:44.712620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:12:55.109673 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:12:55.109637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:13:05.717856 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:13:05.717823 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:13:16.716154 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:13:16.716120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:13:27.406812 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:13:27.406781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:13:38.416740 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:13:38.416707 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:13:47.710368 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:13:47.710331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:13:57.907942 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:13:57.907908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:14:09.005817 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:14:09.005785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:14:18.403480 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:14:18.403427 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:14:28.609287 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:14:28.609245 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:14:53.203339 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:14:53.203260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:15:00.142128 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.142095 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717055-29vsr"] Jul 02 20:15:00.145323 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.145301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" Jul 02 20:15:00.148190 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.148169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-zvdpx\"" Jul 02 20:15:00.161450 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.161428 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717055-29vsr"] Jul 02 20:15:00.236570 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.236542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7jm\" (UniqueName: \"kubernetes.io/projected/fe07c079-de9b-4d66-8442-789d2aa01c1c-kube-api-access-4x7jm\") pod \"maas-api-key-cleanup-29717055-29vsr\" (UID: \"fe07c079-de9b-4d66-8442-789d2aa01c1c\") " pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" Jul 02 20:15:00.310684 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.310652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:15:00.337360 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.337337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7jm\" (UniqueName: \"kubernetes.io/projected/fe07c079-de9b-4d66-8442-789d2aa01c1c-kube-api-access-4x7jm\") pod \"maas-api-key-cleanup-29717055-29vsr\" (UID: \"fe07c079-de9b-4d66-8442-789d2aa01c1c\") " pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" Jul 02 20:15:00.345104 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.345080 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7jm\" (UniqueName: \"kubernetes.io/projected/fe07c079-de9b-4d66-8442-789d2aa01c1c-kube-api-access-4x7jm\") pod \"maas-api-key-cleanup-29717055-29vsr\" (UID: \"fe07c079-de9b-4d66-8442-789d2aa01c1c\") " pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" Jul 02 20:15:00.455751 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.455678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" Jul 02 20:15:00.790895 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.790856 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717055-29vsr"] Jul 02 20:15:00.793138 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:15:00.793104 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe07c079_de9b_4d66_8442_789d2aa01c1c.slice/crio-dc43ef3c7cf26ec3886bbee31d79dd5457b59783a0ae1383ccb2de6bc62dd9a5 WatchSource:0}: Error finding container dc43ef3c7cf26ec3886bbee31d79dd5457b59783a0ae1383ccb2de6bc62dd9a5: Status 404 returned error can't find the container with id dc43ef3c7cf26ec3886bbee31d79dd5457b59783a0ae1383ccb2de6bc62dd9a5 Jul 02 20:15:00.795055 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:00.795037 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:15:01.389868 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:01.389834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" event={"ID":"fe07c079-de9b-4d66-8442-789d2aa01c1c","Type":"ContainerStarted","Data":"dc43ef3c7cf26ec3886bbee31d79dd5457b59783a0ae1383ccb2de6bc62dd9a5"} Jul 02 20:15:05.407958 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:05.407919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" event={"ID":"fe07c079-de9b-4d66-8442-789d2aa01c1c","Type":"ContainerStarted","Data":"239da66cd04418f03c0ef0358dce00d8bf97478cb7c912293d343cd1a9caa79b"} Jul 02 20:15:05.425720 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:05.425673 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" podStartSLOduration=1.5631521689999999 podStartE2EDuration="5.425657163s" totalImagesPullingTime="3.862504994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:05.422941797 +0000 UTC m=+1284.791678522" watchObservedRunningTime="2026-07-02 20:15:05.425657163 +0000 UTC m=+1284.794393889" Jul 02 20:15:06.413379 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:06.413347 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe07c079-de9b-4d66-8442-789d2aa01c1c" containerID="239da66cd04418f03c0ef0358dce00d8bf97478cb7c912293d343cd1a9caa79b" exitCode=0 Jul 02 20:15:06.413775 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:06.413430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" event={"ID":"fe07c079-de9b-4d66-8442-789d2aa01c1c","Type":"ContainerDied","Data":"239da66cd04418f03c0ef0358dce00d8bf97478cb7c912293d343cd1a9caa79b"} Jul 02 20:15:07.541801 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:07.541778 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" Jul 02 20:15:07.704023 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:07.703959 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x7jm\" (UniqueName: \"kubernetes.io/projected/fe07c079-de9b-4d66-8442-789d2aa01c1c-kube-api-access-4x7jm\") pod \"fe07c079-de9b-4d66-8442-789d2aa01c1c\" (UID: \"fe07c079-de9b-4d66-8442-789d2aa01c1c\") " Jul 02 20:15:07.706193 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:07.706167 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe07c079-de9b-4d66-8442-789d2aa01c1c-kube-api-access-4x7jm" (OuterVolumeSpecName: "kube-api-access-4x7jm") pod "fe07c079-de9b-4d66-8442-789d2aa01c1c" (UID: "fe07c079-de9b-4d66-8442-789d2aa01c1c"). InnerVolumeSpecName "kube-api-access-4x7jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:15:07.805003 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:07.804973 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7jm\" (UniqueName: \"kubernetes.io/projected/fe07c079-de9b-4d66-8442-789d2aa01c1c-kube-api-access-4x7jm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:15:08.423998 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:08.423964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" event={"ID":"fe07c079-de9b-4d66-8442-789d2aa01c1c","Type":"ContainerDied","Data":"dc43ef3c7cf26ec3886bbee31d79dd5457b59783a0ae1383ccb2de6bc62dd9a5"} Jul 02 20:15:08.424143 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:08.424012 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc43ef3c7cf26ec3886bbee31d79dd5457b59783a0ae1383ccb2de6bc62dd9a5" Jul 02 20:15:08.424143 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:08.423974 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717055-29vsr" Jul 02 20:15:09.405800 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:09.405768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:15:19.714299 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:19.714264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:15:37.609452 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:37.609418 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:15:48.699976 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:48.699938 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:15:57.302096 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:15:57.302064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:16:06.404729 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:16:06.404692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:16:16.413410 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:16:16.413375 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:16:25.406636 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:16:25.406597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:16:39.713420 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:16:39.713385 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:16:48.910727 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:16:48.910690 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:16:57.805818 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:16:57.805783 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:17:08.405592 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:17:08.405555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:17:18.503256 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:17:18.503218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:17:26.809834 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:17:26.809800 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:17:38.921115 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:17:38.921036 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:17:49.616330 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:17:49.616297 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:18:02.404772 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:18:02.404736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:18:10.904528 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:18:10.904480 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:18:23.503940 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:18:23.503906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:18:31.618772 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:18:31.618733 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:19:13.110097 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:19:13.110062 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:19:21.818933 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:19:21.818901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:20:43.014072 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:20:43.013997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:20:50.103594 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:20:50.103557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:20:56.308539 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:20:56.308503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:21:05.410643 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:21:05.410607 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:21:15.808965 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:21:15.808926 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:21:44.912761 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:21:44.912725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:22:02.087085 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.087055 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw"] Jul 02 20:22:02.087576 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.087401 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe07c079-de9b-4d66-8442-789d2aa01c1c" containerName="cleanup" Jul 02 20:22:02.087576 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.087416 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe07c079-de9b-4d66-8442-789d2aa01c1c" containerName="cleanup" Jul 02 20:22:02.087576 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.087492 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe07c079-de9b-4d66-8442-789d2aa01c1c" containerName="cleanup" Jul 02 20:22:02.090723 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.090695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.094865 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.094838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-28e92a3e\"/\"openshift-service-ca.crt\"" Jul 02 20:22:02.094865 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.094853 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-28e92a3e\"/\"kube-root-ca.crt\"" Jul 02 20:22:02.095026 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.094852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-28e92a3e\"/\"e2e-lifecycle-model-28e92a3e-kserve-self-signed-certs\"" Jul 02 20:22:02.095026 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.094840 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-28e92a3e\"/\"default-dockercfg-m5hgn\"" Jul 02 20:22:02.100949 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.100928 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw"] Jul 02 20:22:02.207668 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.207636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-dshm\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.207668 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.207668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tmp-dir\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.207892 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.207690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-model-cache\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.207892 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.207782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-home\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.207892 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.207842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5q2x\" (UniqueName: \"kubernetes.io/projected/5d1998a7-ee5f-436a-ad13-496aa9b6e957-kube-api-access-v5q2x\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.207892 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.207874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tls-certs\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.308589 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.308560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-home\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.308741 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.308602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5q2x\" (UniqueName: \"kubernetes.io/projected/5d1998a7-ee5f-436a-ad13-496aa9b6e957-kube-api-access-v5q2x\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.308741 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.308625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tls-certs\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.308741 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.308672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-dshm\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.308741 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.308688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tmp-dir\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.308741 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.308707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-model-cache\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.309030 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.309005 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-home\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.309088 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.309034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tmp-dir\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.309135 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.309113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-model-cache\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.310992 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.310963 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-dshm\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.311225 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.311209 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tls-certs\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.316915 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.316897 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5q2x\" (UniqueName: \"kubernetes.io/projected/5d1998a7-ee5f-436a-ad13-496aa9b6e957-kube-api-access-v5q2x\") pod \"e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.400714 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.400687 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:02.525993 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:02.525966 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw"] Jul 02 20:22:02.528523 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:22:02.528495 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d1998a7_ee5f_436a_ad13_496aa9b6e957.slice/crio-f6a1b9078da682c6f00b90314c0204ca8724578f7162ef6f7b400e76198415ba WatchSource:0}: Error finding container f6a1b9078da682c6f00b90314c0204ca8724578f7162ef6f7b400e76198415ba: Status 404 returned error can't find the container with id f6a1b9078da682c6f00b90314c0204ca8724578f7162ef6f7b400e76198415ba Jul 02 20:22:03.187651 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:03.187564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" event={"ID":"5d1998a7-ee5f-436a-ad13-496aa9b6e957","Type":"ContainerStarted","Data":"f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010"} Jul 02 20:22:03.187651 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:03.187604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" event={"ID":"5d1998a7-ee5f-436a-ad13-496aa9b6e957","Type":"ContainerStarted","Data":"f6a1b9078da682c6f00b90314c0204ca8724578f7162ef6f7b400e76198415ba"} Jul 02 20:22:03.204053 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:03.204012 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" podStartSLOduration=1.203997848 podStartE2EDuration="1.203997848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:22:03.202571719 +0000 UTC m=+1702.571308444" watchObservedRunningTime="2026-07-02 20:22:03.203997848 +0000 UTC m=+1702.572734572" Jul 02 20:22:04.904454 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:04.904414 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:22:12.401553 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:12.401519 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:12.401553 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:12.401560 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:12.414092 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:12.414060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:13.241960 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:13.241927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:25.729660 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:25.729629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:22:37.869190 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:37.869145 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw"] Jul 02 20:22:38.321833 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:38.321793 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:22:52.890161 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:52.890120 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" podUID="5d1998a7-ee5f-436a-ad13-496aa9b6e957" containerName="main" containerID="cri-o://f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010" gracePeriod=45 Jul 02 20:22:53.134143 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.134113 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:53.279775 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.279669 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-home\") pod \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " Jul 02 20:22:53.279775 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.279715 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tls-certs\") pod \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " Jul 02 20:22:53.279775 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.279745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-dshm\") pod \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " Jul 02 20:22:53.279775 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.279772 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tmp-dir\") pod \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " Jul 02 20:22:53.280082 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.279796 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5q2x\" (UniqueName: \"kubernetes.io/projected/5d1998a7-ee5f-436a-ad13-496aa9b6e957-kube-api-access-v5q2x\") pod \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " Jul 02 20:22:53.280082 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.279832 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-model-cache\") pod \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\" (UID: \"5d1998a7-ee5f-436a-ad13-496aa9b6e957\") " Jul 02 20:22:53.280082 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.280021 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-home" (OuterVolumeSpecName: "home") pod "5d1998a7-ee5f-436a-ad13-496aa9b6e957" (UID: "5d1998a7-ee5f-436a-ad13-496aa9b6e957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:22:53.280082 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.280040 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d1998a7-ee5f-436a-ad13-496aa9b6e957" (UID: "5d1998a7-ee5f-436a-ad13-496aa9b6e957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:22:53.280277 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.280156 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d1998a7-ee5f-436a-ad13-496aa9b6e957" (UID: "5d1998a7-ee5f-436a-ad13-496aa9b6e957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:22:53.280277 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.280166 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:22:53.280277 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.280185 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:22:53.282065 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.282043 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-dshm" (OuterVolumeSpecName: "dshm") pod "5d1998a7-ee5f-436a-ad13-496aa9b6e957" (UID: "5d1998a7-ee5f-436a-ad13-496aa9b6e957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:22:53.282144 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.282069 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d1998a7-ee5f-436a-ad13-496aa9b6e957" (UID: "5d1998a7-ee5f-436a-ad13-496aa9b6e957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:22:53.282144 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.282131 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1998a7-ee5f-436a-ad13-496aa9b6e957-kube-api-access-v5q2x" (OuterVolumeSpecName: "kube-api-access-v5q2x") pod "5d1998a7-ee5f-436a-ad13-496aa9b6e957" (UID: "5d1998a7-ee5f-436a-ad13-496aa9b6e957"). InnerVolumeSpecName "kube-api-access-v5q2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:22:53.381012 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.380968 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1998a7-ee5f-436a-ad13-496aa9b6e957-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:22:53.381012 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.381003 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:22:53.381012 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.381013 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5q2x\" (UniqueName: \"kubernetes.io/projected/5d1998a7-ee5f-436a-ad13-496aa9b6e957-kube-api-access-v5q2x\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:22:53.381012 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.381021 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1998a7-ee5f-436a-ad13-496aa9b6e957-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:22:53.403794 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.403754 2564 generic.go:356] "Generic (PLEG): container finished" podID="5d1998a7-ee5f-436a-ad13-496aa9b6e957" containerID="f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010" exitCode=0 Jul 02 20:22:53.403950 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.403803 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" Jul 02 20:22:53.403950 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.403834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" event={"ID":"5d1998a7-ee5f-436a-ad13-496aa9b6e957","Type":"ContainerDied","Data":"f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010"} Jul 02 20:22:53.403950 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.403876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw" event={"ID":"5d1998a7-ee5f-436a-ad13-496aa9b6e957","Type":"ContainerDied","Data":"f6a1b9078da682c6f00b90314c0204ca8724578f7162ef6f7b400e76198415ba"} Jul 02 20:22:53.403950 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.403894 2564 scope.go:117] "RemoveContainer" containerID="f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010" Jul 02 20:22:53.412604 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.412583 2564 scope.go:117] "RemoveContainer" containerID="f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010" Jul 02 20:22:53.412870 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:22:53.412854 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010\": container with ID starting with f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010 not found: ID does not exist" containerID="f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010" Jul 02 20:22:53.412922 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.412880 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010"} err="failed to get container status \"f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010\": rpc error: code = NotFound desc = could not find container \"f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010\": container with ID starting with f2c0aa0ce4fa5b8a26f7d82b313d120fc676692c9e5f5ba3ead039f137687010 not found: ID does not exist" Jul 02 20:22:53.426509 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.426478 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw"] Jul 02 20:22:53.428809 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:53.428781 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-28e92a3e/e2e-lifecycle-model-28e92a3e-kserve-5b5c5fcd75-h9xvw"] Jul 02 20:22:55.056411 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:22:55.056375 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1998a7-ee5f-436a-ad13-496aa9b6e957" path="/var/lib/kubelet/pods/5d1998a7-ee5f-436a-ad13-496aa9b6e957/volumes" Jul 02 20:23:17.611204 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:23:17.611169 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:23:38.713589 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:23:38.713556 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:23:46.414348 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:23:46.414313 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:24:31.715299 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:31.715260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:24:43.076514 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:43.076453 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:24:48.819070 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.819041 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h"] Jul 02 20:24:48.819431 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.819344 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1998a7-ee5f-436a-ad13-496aa9b6e957" containerName="main" Jul 02 20:24:48.819431 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.819355 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1998a7-ee5f-436a-ad13-496aa9b6e957" containerName="main" Jul 02 20:24:48.819431 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.819421 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1998a7-ee5f-436a-ad13-496aa9b6e957" containerName="main" Jul 02 20:24:48.822263 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.822244 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.825006 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.824982 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-669041\"/\"openshift-service-ca.crt\"" Jul 02 20:24:48.825133 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.825076 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-669041\"/\"kube-root-ca.crt\"" Jul 02 20:24:48.826319 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.826301 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-669041\"/\"default-dockercfg-454j9\"" Jul 02 20:24:48.826406 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.826323 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-669041\"/\"test-model-669041-kserve-self-signed-certs\"" Jul 02 20:24:48.833411 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.833392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h"] Jul 02 20:24:48.895753 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.895729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-home\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.895867 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.895765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-tmp-dir\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.895867 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.895786 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f94b42d-0373-42e3-84f0-422a7d4b8825-tls-certs\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.895867 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.895827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfqk\" (UniqueName: \"kubernetes.io/projected/9f94b42d-0373-42e3-84f0-422a7d4b8825-kube-api-access-zpfqk\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.895964 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.895898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-model-cache\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.895964 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.895946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-dshm\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.996802 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.996771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-model-cache\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.996962 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.996835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-dshm\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.996962 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.996894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-home\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.996962 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.996926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-tmp-dir\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.996962 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.996954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f94b42d-0373-42e3-84f0-422a7d4b8825-tls-certs\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.997165 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.996979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfqk\" (UniqueName: \"kubernetes.io/projected/9f94b42d-0373-42e3-84f0-422a7d4b8825-kube-api-access-zpfqk\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.997343 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.997312 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-tmp-dir\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.997486 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.997322 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-model-cache\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.997486 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.997365 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-home\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.999114 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.999096 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-dshm\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:48.999273 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:48.999257 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f94b42d-0373-42e3-84f0-422a7d4b8825-tls-certs\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:49.011614 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:49.011597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfqk\" (UniqueName: \"kubernetes.io/projected/9f94b42d-0373-42e3-84f0-422a7d4b8825-kube-api-access-zpfqk\") pod \"test-model-669041-kserve-fd8bf8c9c-28r2h\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:49.132352 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:49.132282 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:49.276812 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:24:49.276778 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f94b42d_0373_42e3_84f0_422a7d4b8825.slice/crio-1ce5fb1c50672cbb38b902524218b30efe257e0a93554b46c1d8cc70934cc7df WatchSource:0}: Error finding container 1ce5fb1c50672cbb38b902524218b30efe257e0a93554b46c1d8cc70934cc7df: Status 404 returned error can't find the container with id 1ce5fb1c50672cbb38b902524218b30efe257e0a93554b46c1d8cc70934cc7df Jul 02 20:24:49.276915 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:49.276876 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h"] Jul 02 20:24:49.908737 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:49.908698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" event={"ID":"9f94b42d-0373-42e3-84f0-422a7d4b8825","Type":"ContainerStarted","Data":"7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7"} Jul 02 20:24:49.909167 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:49.908748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" event={"ID":"9f94b42d-0373-42e3-84f0-422a7d4b8825","Type":"ContainerStarted","Data":"1ce5fb1c50672cbb38b902524218b30efe257e0a93554b46c1d8cc70934cc7df"} Jul 02 20:24:49.931261 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:49.931207 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" podStartSLOduration=1.93118815 podStartE2EDuration="1.93118815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:49.931122266 +0000 UTC m=+1869.299858992" watchObservedRunningTime="2026-07-02 20:24:49.93118815 +0000 UTC m=+1869.299924875" Jul 02 20:24:52.125452 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:52.125413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:24:59.133189 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:59.133155 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:59.133599 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:59.133351 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:59.145792 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:59.145772 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:24:59.963704 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:24:59.963668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:25:22.022381 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:22.022332 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:25:34.509892 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:34.509854 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:25:37.175301 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:37.175260 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h"] Jul 02 20:25:50.214029 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:50.213989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:25:52.195495 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.195416 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" podUID="9f94b42d-0373-42e3-84f0-422a7d4b8825" containerName="main" containerID="cri-o://7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7" gracePeriod=45 Jul 02 20:25:52.443931 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.443904 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:25:52.573306 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573213 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f94b42d-0373-42e3-84f0-422a7d4b8825-tls-certs\") pod \"9f94b42d-0373-42e3-84f0-422a7d4b8825\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " Jul 02 20:25:52.573306 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573258 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-dshm\") pod \"9f94b42d-0373-42e3-84f0-422a7d4b8825\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " Jul 02 20:25:52.573306 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573292 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-tmp-dir\") pod \"9f94b42d-0373-42e3-84f0-422a7d4b8825\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " Jul 02 20:25:52.573306 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573307 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-model-cache\") pod \"9f94b42d-0373-42e3-84f0-422a7d4b8825\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " Jul 02 20:25:52.573676 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpfqk\" (UniqueName: \"kubernetes.io/projected/9f94b42d-0373-42e3-84f0-422a7d4b8825-kube-api-access-zpfqk\") pod \"9f94b42d-0373-42e3-84f0-422a7d4b8825\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " Jul 02 20:25:52.573676 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573382 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-home\") pod \"9f94b42d-0373-42e3-84f0-422a7d4b8825\" (UID: \"9f94b42d-0373-42e3-84f0-422a7d4b8825\") " Jul 02 20:25:52.573676 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573627 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f94b42d-0373-42e3-84f0-422a7d4b8825" (UID: "9f94b42d-0373-42e3-84f0-422a7d4b8825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:25:52.573831 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573802 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-home" (OuterVolumeSpecName: "home") pod "9f94b42d-0373-42e3-84f0-422a7d4b8825" (UID: "9f94b42d-0373-42e3-84f0-422a7d4b8825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:25:52.573893 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.573872 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f94b42d-0373-42e3-84f0-422a7d4b8825" (UID: "9f94b42d-0373-42e3-84f0-422a7d4b8825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:25:52.575639 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.575612 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f94b42d-0373-42e3-84f0-422a7d4b8825-kube-api-access-zpfqk" (OuterVolumeSpecName: "kube-api-access-zpfqk") pod "9f94b42d-0373-42e3-84f0-422a7d4b8825" (UID: "9f94b42d-0373-42e3-84f0-422a7d4b8825"). InnerVolumeSpecName "kube-api-access-zpfqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:25:52.575732 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.575646 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-dshm" (OuterVolumeSpecName: "dshm") pod "9f94b42d-0373-42e3-84f0-422a7d4b8825" (UID: "9f94b42d-0373-42e3-84f0-422a7d4b8825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:25:52.575732 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.575646 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f94b42d-0373-42e3-84f0-422a7d4b8825-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f94b42d-0373-42e3-84f0-422a7d4b8825" (UID: "9f94b42d-0373-42e3-84f0-422a7d4b8825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:25:52.674549 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.674516 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:25:52.674549 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.674546 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f94b42d-0373-42e3-84f0-422a7d4b8825-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:25:52.674549 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.674556 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:25:52.674793 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.674588 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:25:52.674793 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.674597 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f94b42d-0373-42e3-84f0-422a7d4b8825-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:25:52.674793 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:52.674606 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfqk\" (UniqueName: \"kubernetes.io/projected/9f94b42d-0373-42e3-84f0-422a7d4b8825-kube-api-access-zpfqk\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:25:53.183262 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.183229 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f94b42d-0373-42e3-84f0-422a7d4b8825" containerID="7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7" exitCode=0 Jul 02 20:25:53.183531 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.183275 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" Jul 02 20:25:53.183531 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.183320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" event={"ID":"9f94b42d-0373-42e3-84f0-422a7d4b8825","Type":"ContainerDied","Data":"7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7"} Jul 02 20:25:53.183531 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.183367 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h" event={"ID":"9f94b42d-0373-42e3-84f0-422a7d4b8825","Type":"ContainerDied","Data":"1ce5fb1c50672cbb38b902524218b30efe257e0a93554b46c1d8cc70934cc7df"} Jul 02 20:25:53.183531 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.183388 2564 scope.go:117] "RemoveContainer" containerID="7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7" Jul 02 20:25:53.191651 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.191634 2564 scope.go:117] "RemoveContainer" containerID="7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7" Jul 02 20:25:53.191888 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:25:53.191866 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7\": container with ID starting with 7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7 not found: ID does not exist" containerID="7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7" Jul 02 20:25:53.191947 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.191896 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7"} err="failed to get container status \"7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7\": rpc error: code = NotFound desc = could not find container \"7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7\": container with ID starting with 7b4a73fc3f6481eaad68f838cd65bf706711de61fba752482d825d465a6b6bc7 not found: ID does not exist" Jul 02 20:25:53.199162 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.199136 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h"] Jul 02 20:25:53.202707 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:53.202682 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-669041/test-model-669041-kserve-fd8bf8c9c-28r2h"] Jul 02 20:25:55.055648 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:25:55.055610 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f94b42d-0373-42e3-84f0-422a7d4b8825" path="/var/lib/kubelet/pods/9f94b42d-0373-42e3-84f0-422a7d4b8825/volumes" Jul 02 20:26:24.910004 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:26:24.909957 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:26:37.910618 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:26:37.910535 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:26:48.722910 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:26:48.722874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:26:58.662930 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:26:58.662891 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:27:11.837642 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:27:11.837606 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:27:21.944098 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:27:21.944051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:27:36.419825 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:27:36.419790 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:28:00.118237 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.118188 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn"] Jul 02 20:28:00.118785 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.118583 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f94b42d-0373-42e3-84f0-422a7d4b8825" containerName="main" Jul 02 20:28:00.118785 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.118597 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f94b42d-0373-42e3-84f0-422a7d4b8825" containerName="main" Jul 02 20:28:00.118785 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.118675 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f94b42d-0373-42e3-84f0-422a7d4b8825" containerName="main" Jul 02 20:28:00.121652 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.121632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.126681 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:00.126639 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-cd8cf5\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="2acb5eb7-249c-4cb1-a190-1ba56ca46274" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.126681 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:00.126647 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-xrhbf\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-cd8cf5\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"default-dockercfg-xrhbf\"" type="*v1.Secret" Jul 02 20:28:00.126876 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:00.126656 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-cd8cf5\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 02 20:28:00.127047 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:00.127021 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-cd8cf5\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 02 20:28:00.127109 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:00.127040 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"auth-test-model-cd8cf5-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-cd8cf5\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"auth-test-model-cd8cf5-kserve-self-signed-certs\"" type="*v1.Secret" Jul 02 20:28:00.157764 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.157732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn"] Jul 02 20:28:00.231875 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.231835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tmp-dir\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.232050 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.231884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-model-cache\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.232050 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.231908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.232050 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.231959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-dshm\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.232050 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.231992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-home\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.232050 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.232034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfp2h\" (UniqueName: \"kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.332829 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.332792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tmp-dir\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.332829 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.332830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-model-cache\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.333024 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.332849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.333024 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.332970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-dshm\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.333024 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.333014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-home\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.333132 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.333054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfp2h\" (UniqueName: \"kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.333262 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.333236 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-model-cache\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.333303 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.333290 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tmp-dir\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.333445 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.333418 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-home\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:00.335093 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:00.335075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-dshm\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:01.102427 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.102393 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"openshift-service-ca.crt\"" Jul 02 20:28:01.333210 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:01.333165 2564 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Jul 02 20:28:01.333639 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:01.333253 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs podName:2acb5eb7-249c-4cb1-a190-1ba56ca46274 nodeName:}" failed. No retries permitted until 2026-07-02 20:28:01.833233358 +0000 UTC m=+2061.201970061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs") pod "auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274") : failed to sync secret cache: timed out waiting for the condition Jul 02 20:28:01.349614 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:01.349576 2564 projected.go:291] Couldn't get configMap ai-tenant-e2e-shared-a-cd8cf5/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 02 20:28:01.349749 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:01.349624 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dfp2h for pod ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn: failed to sync configmap cache: timed out waiting for the condition Jul 02 20:28:01.349749 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:28:01.349699 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h podName:2acb5eb7-249c-4cb1-a190-1ba56ca46274 nodeName:}" failed. No retries permitted until 2026-07-02 20:28:01.84968061 +0000 UTC m=+2061.218417318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dfp2h" (UniqueName: "kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h") pod "auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274") : failed to sync configmap cache: timed out waiting for the condition Jul 02 20:28:01.421859 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.421833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"default-dockercfg-xrhbf\"" Jul 02 20:28:01.499295 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.499265 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"kube-root-ca.crt\"" Jul 02 20:28:01.632690 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.632652 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"auth-test-model-cd8cf5-kserve-self-signed-certs\"" Jul 02 20:28:01.847519 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.847416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:01.849913 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.849886 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:01.948569 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.948525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfp2h\" (UniqueName: \"kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:01.951068 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:01.951042 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfp2h\" (UniqueName: \"kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h\") pod \"auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:02.230918 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:02.230882 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:02.389333 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:02.389300 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn"] Jul 02 20:28:02.392800 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:28:02.392770 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2acb5eb7_249c_4cb1_a190_1ba56ca46274.slice/crio-76f9003e18d37132aa6a508e61e06d8461ff72ca115006770cb2485d0322bdaf WatchSource:0}: Error finding container 76f9003e18d37132aa6a508e61e06d8461ff72ca115006770cb2485d0322bdaf: Status 404 returned error can't find the container with id 76f9003e18d37132aa6a508e61e06d8461ff72ca115006770cb2485d0322bdaf Jul 02 20:28:02.736219 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:02.736190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" event={"ID":"2acb5eb7-249c-4cb1-a190-1ba56ca46274","Type":"ContainerStarted","Data":"99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235"} Jul 02 20:28:02.736392 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:02.736228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" event={"ID":"2acb5eb7-249c-4cb1-a190-1ba56ca46274","Type":"ContainerStarted","Data":"76f9003e18d37132aa6a508e61e06d8461ff72ca115006770cb2485d0322bdaf"} Jul 02 20:28:02.763684 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:02.763634 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" podStartSLOduration=2.763618342 podStartE2EDuration="2.763618342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:28:02.762576861 +0000 UTC m=+2062.131313586" watchObservedRunningTime="2026-07-02 20:28:02.763618342 +0000 UTC m=+2062.132355066" Jul 02 20:28:03.653990 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:03.653949 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:28:12.231191 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:12.231148 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:12.231858 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:12.231297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:12.243771 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:12.243747 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:12.792740 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:12.792712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:28:31.035236 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.035200 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn"] Jul 02 20:28:31.038567 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.038550 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.043668 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.043636 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"auth-test-model-fa5d1d-kserve-self-signed-certs\"" Jul 02 20:28:31.044647 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.044623 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"openshift-service-ca.crt\"" Jul 02 20:28:31.044917 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.044894 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"default-dockercfg-hmmfs\"" Jul 02 20:28:31.045025 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.044959 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"kube-root-ca.crt\"" Jul 02 20:28:31.065817 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.065778 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn"] Jul 02 20:28:31.120837 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.120794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-model-cache\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.120837 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.120836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-tmp-dir\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.121359 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.121330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-home\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.121506 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.121403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-dshm\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.121769 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.121739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdlb\" (UniqueName: \"kubernetes.io/projected/1d3e06ff-0431-488f-beec-f7f095e0149f-kube-api-access-dvdlb\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.122098 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.122077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3e06ff-0431-488f-beec-f7f095e0149f-tls-certs\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223056 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-home\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223056 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-dshm\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223270 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdlb\" (UniqueName: \"kubernetes.io/projected/1d3e06ff-0431-488f-beec-f7f095e0149f-kube-api-access-dvdlb\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223270 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3e06ff-0431-488f-beec-f7f095e0149f-tls-certs\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223369 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-model-cache\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223422 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-tmp-dir\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223580 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223549 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-home\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223702 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223684 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-tmp-dir\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.223794 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.223772 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-model-cache\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.225340 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.225313 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-dshm\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.225646 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.225623 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3e06ff-0431-488f-beec-f7f095e0149f-tls-certs\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.233783 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.233752 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdlb\" (UniqueName: \"kubernetes.io/projected/1d3e06ff-0431-488f-beec-f7f095e0149f-kube-api-access-dvdlb\") pod \"auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.348439 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.348345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:31.493356 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.493198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn"] Jul 02 20:28:31.496336 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:28:31.496293 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d3e06ff_0431_488f_beec_f7f095e0149f.slice/crio-e425a3cb5e8c403cf235a3aaae91d834209638dcf35c9926515fb93a3e3464bf WatchSource:0}: Error finding container e425a3cb5e8c403cf235a3aaae91d834209638dcf35c9926515fb93a3e3464bf: Status 404 returned error can't find the container with id e425a3cb5e8c403cf235a3aaae91d834209638dcf35c9926515fb93a3e3464bf Jul 02 20:28:31.865313 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.865277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" event={"ID":"1d3e06ff-0431-488f-beec-f7f095e0149f","Type":"ContainerStarted","Data":"9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86"} Jul 02 20:28:31.865532 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.865320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" event={"ID":"1d3e06ff-0431-488f-beec-f7f095e0149f","Type":"ContainerStarted","Data":"e425a3cb5e8c403cf235a3aaae91d834209638dcf35c9926515fb93a3e3464bf"} Jul 02 20:28:31.905443 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:31.905384 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" podStartSLOduration=0.905368009 podStartE2EDuration="905.368009ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:28:31.90388635 +0000 UTC m=+2091.272623076" watchObservedRunningTime="2026-07-02 20:28:31.905368009 +0000 UTC m=+2091.274104733" Jul 02 20:28:34.633045 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:34.633004 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:28:41.348560 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:41.348508 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:41.349037 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:41.348579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:41.361530 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:41.361504 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:41.921158 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:41.921125 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:28:48.327097 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:28:48.327058 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:29:03.742563 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:29:03.742527 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:29:18.828854 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:29:18.828819 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:29:34.513244 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:29:34.513166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:29:47.619955 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:29:47.619918 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:30:00.132211 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.132176 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717070-gs7x4"] Jul 02 20:30:00.135532 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.135510 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr"] Jul 02 20:30:00.135686 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.135632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" Jul 02 20:30:00.138561 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.138539 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-zvdpx\"" Jul 02 20:30:00.139386 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.139365 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" Jul 02 20:30:00.143180 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.143160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717070-gs7x4"] Jul 02 20:30:00.145898 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.145876 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr"] Jul 02 20:30:00.227736 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.227703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69cb\" (UniqueName: \"kubernetes.io/projected/e2fb8c8a-b798-42d3-976b-624ab5b65a66-kube-api-access-k69cb\") pod \"maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr\" (UID: \"e2fb8c8a-b798-42d3-976b-624ab5b65a66\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" Jul 02 20:30:00.227880 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.227810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56lp\" (UniqueName: \"kubernetes.io/projected/20fa5d84-5e2d-41b3-ba4d-7928c820ff24-kube-api-access-w56lp\") pod \"maas-api-key-cleanup-29717070-gs7x4\" (UID: \"20fa5d84-5e2d-41b3-ba4d-7928c820ff24\") " pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" Jul 02 20:30:00.328437 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.328406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69cb\" (UniqueName: \"kubernetes.io/projected/e2fb8c8a-b798-42d3-976b-624ab5b65a66-kube-api-access-k69cb\") pod \"maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr\" (UID: \"e2fb8c8a-b798-42d3-976b-624ab5b65a66\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" Jul 02 20:30:00.328589 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.328495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56lp\" (UniqueName: \"kubernetes.io/projected/20fa5d84-5e2d-41b3-ba4d-7928c820ff24-kube-api-access-w56lp\") pod \"maas-api-key-cleanup-29717070-gs7x4\" (UID: \"20fa5d84-5e2d-41b3-ba4d-7928c820ff24\") " pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" Jul 02 20:30:00.337824 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.337795 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69cb\" (UniqueName: \"kubernetes.io/projected/e2fb8c8a-b798-42d3-976b-624ab5b65a66-kube-api-access-k69cb\") pod \"maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr\" (UID: \"e2fb8c8a-b798-42d3-976b-624ab5b65a66\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" Jul 02 20:30:00.337931 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.337895 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56lp\" (UniqueName: \"kubernetes.io/projected/20fa5d84-5e2d-41b3-ba4d-7928c820ff24-kube-api-access-w56lp\") pod \"maas-api-key-cleanup-29717070-gs7x4\" (UID: \"20fa5d84-5e2d-41b3-ba4d-7928c820ff24\") " pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" Jul 02 20:30:00.449322 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.449286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" Jul 02 20:30:00.456349 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.456331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" Jul 02 20:30:00.615162 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.615137 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr"] Jul 02 20:30:00.617761 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:30:00.617729 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fb8c8a_b798_42d3_976b_624ab5b65a66.slice/crio-e232a79d46a9431f20ce14fdba3b1ab8d704987ed26cccef22e8de6240580ef1 WatchSource:0}: Error finding container e232a79d46a9431f20ce14fdba3b1ab8d704987ed26cccef22e8de6240580ef1: Status 404 returned error can't find the container with id e232a79d46a9431f20ce14fdba3b1ab8d704987ed26cccef22e8de6240580ef1 Jul 02 20:30:00.799832 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:00.799800 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717070-gs7x4"] Jul 02 20:30:00.861383 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:30:00.861343 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fa5d84_5e2d_41b3_ba4d_7928c820ff24.slice/crio-07f2943b06bb75d92776c7e0b70d00cbdaedd1d467256ad0fc78c074ecfad97e WatchSource:0}: Error finding container 07f2943b06bb75d92776c7e0b70d00cbdaedd1d467256ad0fc78c074ecfad97e: Status 404 returned error can't find the container with id 07f2943b06bb75d92776c7e0b70d00cbdaedd1d467256ad0fc78c074ecfad97e Jul 02 20:30:01.267954 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:01.267917 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" event={"ID":"20fa5d84-5e2d-41b3-ba4d-7928c820ff24","Type":"ContainerStarted","Data":"505e3da41e9dc457df6401ce976abb9bb415edcafceb1cbbbf11e17eecbfdc1b"} Jul 02 20:30:01.268398 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:01.267979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" event={"ID":"20fa5d84-5e2d-41b3-ba4d-7928c820ff24","Type":"ContainerStarted","Data":"07f2943b06bb75d92776c7e0b70d00cbdaedd1d467256ad0fc78c074ecfad97e"} Jul 02 20:30:01.269302 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:01.269278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" event={"ID":"e2fb8c8a-b798-42d3-976b-624ab5b65a66","Type":"ContainerStarted","Data":"a41bb71143ebd09df6ec0d795f0f57ec2a9df68164d669346b71a7ce61b425b6"} Jul 02 20:30:01.269412 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:01.269311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" event={"ID":"e2fb8c8a-b798-42d3-976b-624ab5b65a66","Type":"ContainerStarted","Data":"e232a79d46a9431f20ce14fdba3b1ab8d704987ed26cccef22e8de6240580ef1"} Jul 02 20:30:01.285328 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:01.285282 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" podStartSLOduration=1.285269226 podStartE2EDuration="1.285269226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:30:01.284266336 +0000 UTC m=+2180.653003061" watchObservedRunningTime="2026-07-02 20:30:01.285269226 +0000 UTC m=+2180.654005951" Jul 02 20:30:01.301923 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:01.301878 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" podStartSLOduration=1.301861184 podStartE2EDuration="1.301861184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:30:01.300646123 +0000 UTC m=+2180.669382848" watchObservedRunningTime="2026-07-02 20:30:01.301861184 +0000 UTC m=+2180.670597910" Jul 02 20:30:02.275418 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:02.275380 2564 generic.go:356] "Generic (PLEG): container finished" podID="20fa5d84-5e2d-41b3-ba4d-7928c820ff24" containerID="505e3da41e9dc457df6401ce976abb9bb415edcafceb1cbbbf11e17eecbfdc1b" exitCode=0 Jul 02 20:30:02.275875 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:02.275485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" event={"ID":"20fa5d84-5e2d-41b3-ba4d-7928c820ff24","Type":"ContainerDied","Data":"505e3da41e9dc457df6401ce976abb9bb415edcafceb1cbbbf11e17eecbfdc1b"} Jul 02 20:30:02.276926 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:02.276902 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2fb8c8a-b798-42d3-976b-624ab5b65a66" containerID="a41bb71143ebd09df6ec0d795f0f57ec2a9df68164d669346b71a7ce61b425b6" exitCode=0 Jul 02 20:30:02.277026 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:02.276943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" event={"ID":"e2fb8c8a-b798-42d3-976b-624ab5b65a66","Type":"ContainerDied","Data":"a41bb71143ebd09df6ec0d795f0f57ec2a9df68164d669346b71a7ce61b425b6"} Jul 02 20:30:03.423375 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.423351 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" Jul 02 20:30:03.426339 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.426318 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" Jul 02 20:30:03.559962 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.559874 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69cb\" (UniqueName: \"kubernetes.io/projected/e2fb8c8a-b798-42d3-976b-624ab5b65a66-kube-api-access-k69cb\") pod \"e2fb8c8a-b798-42d3-976b-624ab5b65a66\" (UID: \"e2fb8c8a-b798-42d3-976b-624ab5b65a66\") " Jul 02 20:30:03.559962 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.559918 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w56lp\" (UniqueName: \"kubernetes.io/projected/20fa5d84-5e2d-41b3-ba4d-7928c820ff24-kube-api-access-w56lp\") pod \"20fa5d84-5e2d-41b3-ba4d-7928c820ff24\" (UID: \"20fa5d84-5e2d-41b3-ba4d-7928c820ff24\") " Jul 02 20:30:03.562064 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.562028 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fb8c8a-b798-42d3-976b-624ab5b65a66-kube-api-access-k69cb" (OuterVolumeSpecName: "kube-api-access-k69cb") pod "e2fb8c8a-b798-42d3-976b-624ab5b65a66" (UID: "e2fb8c8a-b798-42d3-976b-624ab5b65a66"). InnerVolumeSpecName "kube-api-access-k69cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:30:03.562176 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.562064 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fa5d84-5e2d-41b3-ba4d-7928c820ff24-kube-api-access-w56lp" (OuterVolumeSpecName: "kube-api-access-w56lp") pod "20fa5d84-5e2d-41b3-ba4d-7928c820ff24" (UID: "20fa5d84-5e2d-41b3-ba4d-7928c820ff24"). InnerVolumeSpecName "kube-api-access-w56lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:30:03.661529 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.661498 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69cb\" (UniqueName: \"kubernetes.io/projected/e2fb8c8a-b798-42d3-976b-624ab5b65a66-kube-api-access-k69cb\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:03.661529 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:03.661525 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56lp\" (UniqueName: \"kubernetes.io/projected/20fa5d84-5e2d-41b3-ba4d-7928c820ff24-kube-api-access-w56lp\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:04.011597 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:04.011563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:30:04.287729 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:04.287651 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" Jul 02 20:30:04.287729 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:04.287690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717070-gs7x4" event={"ID":"20fa5d84-5e2d-41b3-ba4d-7928c820ff24","Type":"ContainerDied","Data":"07f2943b06bb75d92776c7e0b70d00cbdaedd1d467256ad0fc78c074ecfad97e"} Jul 02 20:30:04.287933 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:04.287736 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07f2943b06bb75d92776c7e0b70d00cbdaedd1d467256ad0fc78c074ecfad97e" Jul 02 20:30:04.288854 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:04.288830 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" Jul 02 20:30:04.288958 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:04.288833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr" event={"ID":"e2fb8c8a-b798-42d3-976b-624ab5b65a66","Type":"ContainerDied","Data":"e232a79d46a9431f20ce14fdba3b1ab8d704987ed26cccef22e8de6240580ef1"} Jul 02 20:30:04.289015 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:04.288960 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e232a79d46a9431f20ce14fdba3b1ab8d704987ed26cccef22e8de6240580ef1" Jul 02 20:30:19.714523 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:19.714490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:30:22.863677 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:22.863642 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn"] Jul 02 20:30:23.776357 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:23.776325 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn"] Jul 02 20:30:33.189378 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.189346 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66"] Jul 02 20:30:33.189869 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.189707 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fa5d84-5e2d-41b3-ba4d-7928c820ff24" containerName="cleanup" Jul 02 20:30:33.189869 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.189720 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fa5d84-5e2d-41b3-ba4d-7928c820ff24" containerName="cleanup" Jul 02 20:30:33.189869 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.189739 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fb8c8a-b798-42d3-976b-624ab5b65a66" containerName="cleanup" Jul 02 20:30:33.189869 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.189745 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fb8c8a-b798-42d3-976b-624ab5b65a66" containerName="cleanup" Jul 02 20:30:33.189869 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.189791 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fa5d84-5e2d-41b3-ba4d-7928c820ff24" containerName="cleanup" Jul 02 20:30:33.189869 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.189803 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2fb8c8a-b798-42d3-976b-624ab5b65a66" containerName="cleanup" Jul 02 20:30:33.192973 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.192958 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.195682 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.195661 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cd8cf5\"/\"sub-test-model-cd8cf5-kserve-self-signed-certs\"" Jul 02 20:30:33.204160 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.204139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66"] Jul 02 20:30:33.222708 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.222685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-home\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.222851 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.222725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-dshm\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.222851 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.222750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-model-cache\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.222928 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.222846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-tmp-dir\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.222928 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.222876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe39959-7658-4a17-a52d-45401a4cb657-tls-certs\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.222928 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.222903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnghp\" (UniqueName: \"kubernetes.io/projected/7fe39959-7658-4a17-a52d-45401a4cb657-kube-api-access-jnghp\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324339 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-home\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324339 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-dshm\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-model-cache\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-tmp-dir\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe39959-7658-4a17-a52d-45401a4cb657-tls-certs\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnghp\" (UniqueName: \"kubernetes.io/projected/7fe39959-7658-4a17-a52d-45401a4cb657-kube-api-access-jnghp\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324813 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324709 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-home\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324813 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324761 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-model-cache\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.324920 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.324818 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-tmp-dir\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.326755 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.326733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-dshm\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.326909 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.326889 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe39959-7658-4a17-a52d-45401a4cb657-tls-certs\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.333200 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.333180 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnghp\" (UniqueName: \"kubernetes.io/projected/7fe39959-7658-4a17-a52d-45401a4cb657-kube-api-access-jnghp\") pod \"sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.502296 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.502203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:33.631495 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:33.631445 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66"] Jul 02 20:30:33.634214 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:30:33.634180 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fe39959_7658_4a17_a52d_45401a4cb657.slice/crio-9ea57a4ec942d0c61f10b6e18b490bcc7d00136569d1b87e61d3b61baa275f17 WatchSource:0}: Error finding container 9ea57a4ec942d0c61f10b6e18b490bcc7d00136569d1b87e61d3b61baa275f17: Status 404 returned error can't find the container with id 9ea57a4ec942d0c61f10b6e18b490bcc7d00136569d1b87e61d3b61baa275f17 Jul 02 20:30:34.421013 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:34.420971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" event={"ID":"7fe39959-7658-4a17-a52d-45401a4cb657","Type":"ContainerStarted","Data":"c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317"} Jul 02 20:30:34.421381 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:34.421021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" event={"ID":"7fe39959-7658-4a17-a52d-45401a4cb657","Type":"ContainerStarted","Data":"9ea57a4ec942d0c61f10b6e18b490bcc7d00136569d1b87e61d3b61baa275f17"} Jul 02 20:30:34.440413 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:34.440374 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" podStartSLOduration=1.440360118 podStartE2EDuration="1.440360118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:30:34.439514999 +0000 UTC m=+2213.808251721" watchObservedRunningTime="2026-07-02 20:30:34.440360118 +0000 UTC m=+2213.809096851" Jul 02 20:30:36.311948 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:36.311914 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:30:37.883490 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:37.883421 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" podUID="2acb5eb7-249c-4cb1-a190-1ba56ca46274" containerName="main" containerID="cri-o://99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235" gracePeriod=45 Jul 02 20:30:38.137032 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.136962 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:30:38.271120 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271079 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs\") pod \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " Jul 02 20:30:38.271120 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271133 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tmp-dir\") pod \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " Jul 02 20:30:38.271359 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271179 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-dshm\") pod \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " Jul 02 20:30:38.271359 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271197 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-model-cache\") pod \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " Jul 02 20:30:38.271359 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271214 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-home\") pod \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " Jul 02 20:30:38.271359 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271245 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfp2h\" (UniqueName: \"kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h\") pod \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\" (UID: \"2acb5eb7-249c-4cb1-a190-1ba56ca46274\") " Jul 02 20:30:38.271596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271437 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-model-cache" (OuterVolumeSpecName: "model-cache") pod "2acb5eb7-249c-4cb1-a190-1ba56ca46274" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:38.271596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271492 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2acb5eb7-249c-4cb1-a190-1ba56ca46274" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:38.271596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271539 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-home" (OuterVolumeSpecName: "home") pod "2acb5eb7-249c-4cb1-a190-1ba56ca46274" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:38.271596 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.271570 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:38.273387 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.273355 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-dshm" (OuterVolumeSpecName: "dshm") pod "2acb5eb7-249c-4cb1-a190-1ba56ca46274" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:38.273387 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.273371 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2acb5eb7-249c-4cb1-a190-1ba56ca46274" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:30:38.273521 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.273404 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h" (OuterVolumeSpecName: "kube-api-access-dfp2h") pod "2acb5eb7-249c-4cb1-a190-1ba56ca46274" (UID: "2acb5eb7-249c-4cb1-a190-1ba56ca46274"). InnerVolumeSpecName "kube-api-access-dfp2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:30:38.372147 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.372117 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:38.372147 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.372143 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:38.372147 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.372153 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2acb5eb7-249c-4cb1-a190-1ba56ca46274-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:38.372337 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.372162 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfp2h\" (UniqueName: \"kubernetes.io/projected/2acb5eb7-249c-4cb1-a190-1ba56ca46274-kube-api-access-dfp2h\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:38.372337 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.372171 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2acb5eb7-249c-4cb1-a190-1ba56ca46274-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:38.440598 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.440564 2564 generic.go:356] "Generic (PLEG): container finished" podID="2acb5eb7-249c-4cb1-a190-1ba56ca46274" containerID="99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235" exitCode=0 Jul 02 20:30:38.440767 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.440616 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" Jul 02 20:30:38.440767 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.440658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" event={"ID":"2acb5eb7-249c-4cb1-a190-1ba56ca46274","Type":"ContainerDied","Data":"99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235"} Jul 02 20:30:38.440767 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.440699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn" event={"ID":"2acb5eb7-249c-4cb1-a190-1ba56ca46274","Type":"ContainerDied","Data":"76f9003e18d37132aa6a508e61e06d8461ff72ca115006770cb2485d0322bdaf"} Jul 02 20:30:38.440767 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.440719 2564 scope.go:117] "RemoveContainer" containerID="99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235" Jul 02 20:30:38.449436 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.449419 2564 scope.go:117] "RemoveContainer" containerID="99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235" Jul 02 20:30:38.449716 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:30:38.449698 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235\": container with ID starting with 99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235 not found: ID does not exist" containerID="99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235" Jul 02 20:30:38.449769 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.449724 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235"} err="failed to get container status \"99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235\": rpc error: code = NotFound desc = could not find container \"99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235\": container with ID starting with 99421d04131280b98da3e21bfa2b4b48fc71745c5506e6fb1ad915a8732d4235 not found: ID does not exist" Jul 02 20:30:38.462950 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.462927 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn"] Jul 02 20:30:38.469749 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.469727 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/auth-test-model-cd8cf5-kserve-6df5787b4c-8sxzn"] Jul 02 20:30:38.796501 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:38.796396 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" podUID="1d3e06ff-0431-488f-beec-f7f095e0149f" containerName="main" containerID="cri-o://9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86" gracePeriod=45 Jul 02 20:30:39.050806 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.050734 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:30:39.053834 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.053808 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2acb5eb7-249c-4cb1-a190-1ba56ca46274" path="/var/lib/kubelet/pods/2acb5eb7-249c-4cb1-a190-1ba56ca46274/volumes" Jul 02 20:30:39.180053 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180026 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-model-cache\") pod \"1d3e06ff-0431-488f-beec-f7f095e0149f\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " Jul 02 20:30:39.180053 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-home\") pod \"1d3e06ff-0431-488f-beec-f7f095e0149f\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " Jul 02 20:30:39.180256 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180077 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvdlb\" (UniqueName: \"kubernetes.io/projected/1d3e06ff-0431-488f-beec-f7f095e0149f-kube-api-access-dvdlb\") pod \"1d3e06ff-0431-488f-beec-f7f095e0149f\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " Jul 02 20:30:39.180256 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180145 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3e06ff-0431-488f-beec-f7f095e0149f-tls-certs\") pod \"1d3e06ff-0431-488f-beec-f7f095e0149f\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " Jul 02 20:30:39.180256 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-dshm\") pod \"1d3e06ff-0431-488f-beec-f7f095e0149f\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " Jul 02 20:30:39.180256 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180221 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-tmp-dir\") pod \"1d3e06ff-0431-488f-beec-f7f095e0149f\" (UID: \"1d3e06ff-0431-488f-beec-f7f095e0149f\") " Jul 02 20:30:39.180497 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180334 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d3e06ff-0431-488f-beec-f7f095e0149f" (UID: "1d3e06ff-0431-488f-beec-f7f095e0149f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:39.180497 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180340 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-home" (OuterVolumeSpecName: "home") pod "1d3e06ff-0431-488f-beec-f7f095e0149f" (UID: "1d3e06ff-0431-488f-beec-f7f095e0149f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:39.180497 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180479 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:39.180497 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180497 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:39.180678 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.180555 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d3e06ff-0431-488f-beec-f7f095e0149f" (UID: "1d3e06ff-0431-488f-beec-f7f095e0149f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:39.182244 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.182216 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-dshm" (OuterVolumeSpecName: "dshm") pod "1d3e06ff-0431-488f-beec-f7f095e0149f" (UID: "1d3e06ff-0431-488f-beec-f7f095e0149f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:30:39.182351 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.182311 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3e06ff-0431-488f-beec-f7f095e0149f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d3e06ff-0431-488f-beec-f7f095e0149f" (UID: "1d3e06ff-0431-488f-beec-f7f095e0149f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:30:39.182401 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.182341 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3e06ff-0431-488f-beec-f7f095e0149f-kube-api-access-dvdlb" (OuterVolumeSpecName: "kube-api-access-dvdlb") pod "1d3e06ff-0431-488f-beec-f7f095e0149f" (UID: "1d3e06ff-0431-488f-beec-f7f095e0149f"). InnerVolumeSpecName "kube-api-access-dvdlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:30:39.281606 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.281571 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:39.281606 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.281602 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e06ff-0431-488f-beec-f7f095e0149f-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:39.281794 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.281617 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdlb\" (UniqueName: \"kubernetes.io/projected/1d3e06ff-0431-488f-beec-f7f095e0149f-kube-api-access-dvdlb\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:39.281794 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.281629 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3e06ff-0431-488f-beec-f7f095e0149f-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:30:39.447388 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.447352 2564 generic.go:356] "Generic (PLEG): container finished" podID="1d3e06ff-0431-488f-beec-f7f095e0149f" containerID="9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86" exitCode=0 Jul 02 20:30:39.447573 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.447417 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" Jul 02 20:30:39.447573 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.447420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" event={"ID":"1d3e06ff-0431-488f-beec-f7f095e0149f","Type":"ContainerDied","Data":"9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86"} Jul 02 20:30:39.447573 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.447484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn" event={"ID":"1d3e06ff-0431-488f-beec-f7f095e0149f","Type":"ContainerDied","Data":"e425a3cb5e8c403cf235a3aaae91d834209638dcf35c9926515fb93a3e3464bf"} Jul 02 20:30:39.447573 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.447508 2564 scope.go:117] "RemoveContainer" containerID="9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86" Jul 02 20:30:39.456382 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.456365 2564 scope.go:117] "RemoveContainer" containerID="9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86" Jul 02 20:30:39.456675 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:30:39.456656 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86\": container with ID starting with 9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86 not found: ID does not exist" containerID="9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86" Jul 02 20:30:39.456724 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.456683 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86"} err="failed to get container status \"9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86\": rpc error: code = NotFound desc = could not find container \"9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86\": container with ID starting with 9e5ce81f2893f62fdb146c877168b1b5a588cd8587d933d2ac43a6d4baea2e86 not found: ID does not exist" Jul 02 20:30:39.472086 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.472061 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn"] Jul 02 20:30:39.476807 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:39.476782 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/auth-test-model-fa5d1d-kserve-5655dc6799-xvzkn"] Jul 02 20:30:41.054252 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:41.054221 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d3e06ff-0431-488f-beec-f7f095e0149f" path="/var/lib/kubelet/pods/1d3e06ff-0431-488f-beec-f7f095e0149f/volumes" Jul 02 20:30:43.502434 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:43.502402 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:43.502434 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:43.502442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:43.514982 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:43.514956 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:44.481931 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:44.481901 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:30:54.893040 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.893006 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8"] Jul 02 20:30:54.893425 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.893349 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2acb5eb7-249c-4cb1-a190-1ba56ca46274" containerName="main" Jul 02 20:30:54.893425 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.893359 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2acb5eb7-249c-4cb1-a190-1ba56ca46274" containerName="main" Jul 02 20:30:54.893425 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.893378 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3e06ff-0431-488f-beec-f7f095e0149f" containerName="main" Jul 02 20:30:54.893425 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.893384 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3e06ff-0431-488f-beec-f7f095e0149f" containerName="main" Jul 02 20:30:54.893586 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.893435 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3e06ff-0431-488f-beec-f7f095e0149f" containerName="main" Jul 02 20:30:54.893586 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.893445 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2acb5eb7-249c-4cb1-a190-1ba56ca46274" containerName="main" Jul 02 20:30:54.896518 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.896500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:54.900483 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.900441 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"kube-root-ca.crt\"" Jul 02 20:30:54.900678 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.900653 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"default-dockercfg-hmmfs\"" Jul 02 20:30:54.900797 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.900660 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"sub-test-model-fa5d1d-kserve-self-signed-certs\"" Jul 02 20:30:54.900797 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.900688 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"openshift-service-ca.crt\"" Jul 02 20:30:54.912699 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:54.912678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8"] Jul 02 20:30:55.017703 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.017668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-tmp-dir\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.017851 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.017744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-dshm\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.017851 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.017843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-home\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.017927 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.017867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-model-cache\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.017963 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.017941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f746f-0999-4e43-9b7f-03c069206b82-tls-certs\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.017995 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.017966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg4p6\" (UniqueName: \"kubernetes.io/projected/704f746f-0999-4e43-9b7f-03c069206b82-kube-api-access-kg4p6\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119192 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-home\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119192 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-model-cache\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119423 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f746f-0999-4e43-9b7f-03c069206b82-tls-certs\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119423 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg4p6\" (UniqueName: \"kubernetes.io/projected/704f746f-0999-4e43-9b7f-03c069206b82-kube-api-access-kg4p6\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119423 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-tmp-dir\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119597 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-dshm\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119597 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119577 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-model-cache\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119702 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119640 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-home\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.119765 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.119745 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-tmp-dir\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.121683 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.121663 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-dshm\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.121800 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.121784 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f746f-0999-4e43-9b7f-03c069206b82-tls-certs\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.127571 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.127543 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg4p6\" (UniqueName: \"kubernetes.io/projected/704f746f-0999-4e43-9b7f-03c069206b82-kube-api-access-kg4p6\") pod \"sub-test-model-fa5d1d-kserve-74b56769-bvdj8\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.206124 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.206039 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:30:55.334644 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.334620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8"] Jul 02 20:30:55.336668 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:30:55.336641 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod704f746f_0999_4e43_9b7f_03c069206b82.slice/crio-9424f40b6ca9ddaeb11d645519174c86722a5c91c82d1ed90f92d093b11b0982 WatchSource:0}: Error finding container 9424f40b6ca9ddaeb11d645519174c86722a5c91c82d1ed90f92d093b11b0982: Status 404 returned error can't find the container with id 9424f40b6ca9ddaeb11d645519174c86722a5c91c82d1ed90f92d093b11b0982 Jul 02 20:30:55.524264 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.524174 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" event={"ID":"704f746f-0999-4e43-9b7f-03c069206b82","Type":"ContainerStarted","Data":"569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682"} Jul 02 20:30:55.524264 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.524214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" event={"ID":"704f746f-0999-4e43-9b7f-03c069206b82","Type":"ContainerStarted","Data":"9424f40b6ca9ddaeb11d645519174c86722a5c91c82d1ed90f92d093b11b0982"} Jul 02 20:30:55.545436 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:55.545388 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" podStartSLOduration=1.5453728770000001 podStartE2EDuration="1.545372877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:30:55.544169064 +0000 UTC m=+2234.912905790" watchObservedRunningTime="2026-07-02 20:30:55.545372877 +0000 UTC m=+2234.914109601" Jul 02 20:30:59.108534 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:30:59.108495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:31:05.206293 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:05.206251 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:31:05.206937 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:05.206308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:31:05.218875 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:05.218848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:31:05.585963 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:05.585869 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:31:28.907721 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:28.907684 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:31:46.008548 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:46.008506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:31:52.914903 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:52.914862 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66"] Jul 02 20:31:53.568198 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:53.568161 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8"] Jul 02 20:31:59.802744 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:31:59.802699 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:32:07.935562 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:07.935515 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" podUID="7fe39959-7658-4a17-a52d-45401a4cb657" containerName="main" containerID="cri-o://c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317" gracePeriod=45 Jul 02 20:32:08.184844 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.184820 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:32:08.255714 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.255625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe39959-7658-4a17-a52d-45401a4cb657-tls-certs\") pod \"7fe39959-7658-4a17-a52d-45401a4cb657\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " Jul 02 20:32:08.255714 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.255707 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnghp\" (UniqueName: \"kubernetes.io/projected/7fe39959-7658-4a17-a52d-45401a4cb657-kube-api-access-jnghp\") pod \"7fe39959-7658-4a17-a52d-45401a4cb657\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " Jul 02 20:32:08.255945 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.255744 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-dshm\") pod \"7fe39959-7658-4a17-a52d-45401a4cb657\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " Jul 02 20:32:08.255945 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.255764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-tmp-dir\") pod \"7fe39959-7658-4a17-a52d-45401a4cb657\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " Jul 02 20:32:08.255945 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.255822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-model-cache\") pod \"7fe39959-7658-4a17-a52d-45401a4cb657\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " Jul 02 20:32:08.255945 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.255848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-home\") pod \"7fe39959-7658-4a17-a52d-45401a4cb657\" (UID: \"7fe39959-7658-4a17-a52d-45401a4cb657\") " Jul 02 20:32:08.256142 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.256052 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fe39959-7658-4a17-a52d-45401a4cb657" (UID: "7fe39959-7658-4a17-a52d-45401a4cb657"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.256200 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.256146 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fe39959-7658-4a17-a52d-45401a4cb657" (UID: "7fe39959-7658-4a17-a52d-45401a4cb657"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.256200 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.256163 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.256200 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.256167 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-home" (OuterVolumeSpecName: "home") pod "7fe39959-7658-4a17-a52d-45401a4cb657" (UID: "7fe39959-7658-4a17-a52d-45401a4cb657"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.257816 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.257795 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe39959-7658-4a17-a52d-45401a4cb657-kube-api-access-jnghp" (OuterVolumeSpecName: "kube-api-access-jnghp") pod "7fe39959-7658-4a17-a52d-45401a4cb657" (UID: "7fe39959-7658-4a17-a52d-45401a4cb657"). InnerVolumeSpecName "kube-api-access-jnghp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:32:08.258239 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.258224 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe39959-7658-4a17-a52d-45401a4cb657-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fe39959-7658-4a17-a52d-45401a4cb657" (UID: "7fe39959-7658-4a17-a52d-45401a4cb657"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:32:08.258310 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.258235 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-dshm" (OuterVolumeSpecName: "dshm") pod "7fe39959-7658-4a17-a52d-45401a4cb657" (UID: "7fe39959-7658-4a17-a52d-45401a4cb657"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.356857 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.356821 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnghp\" (UniqueName: \"kubernetes.io/projected/7fe39959-7658-4a17-a52d-45401a4cb657-kube-api-access-jnghp\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.356857 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.356852 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.356857 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.356863 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.357091 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.356872 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe39959-7658-4a17-a52d-45401a4cb657-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.357091 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.356880 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe39959-7658-4a17-a52d-45401a4cb657-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.589373 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.589265 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" podUID="704f746f-0999-4e43-9b7f-03c069206b82" containerName="main" containerID="cri-o://569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682" gracePeriod=45 Jul 02 20:32:08.828914 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.828889 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:32:08.861344 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861264 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-tmp-dir\") pod \"704f746f-0999-4e43-9b7f-03c069206b82\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " Jul 02 20:32:08.861344 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861319 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg4p6\" (UniqueName: \"kubernetes.io/projected/704f746f-0999-4e43-9b7f-03c069206b82-kube-api-access-kg4p6\") pod \"704f746f-0999-4e43-9b7f-03c069206b82\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " Jul 02 20:32:08.861614 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-model-cache\") pod \"704f746f-0999-4e43-9b7f-03c069206b82\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " Jul 02 20:32:08.861614 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861485 2564 generic.go:356] "Generic (PLEG): container finished" podID="704f746f-0999-4e43-9b7f-03c069206b82" containerID="569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682" exitCode=0 Jul 02 20:32:08.861614 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-dshm\") pod \"704f746f-0999-4e43-9b7f-03c069206b82\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " Jul 02 20:32:08.861614 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861545 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" Jul 02 20:32:08.861614 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861601 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "704f746f-0999-4e43-9b7f-03c069206b82" (UID: "704f746f-0999-4e43-9b7f-03c069206b82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.861614 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" event={"ID":"704f746f-0999-4e43-9b7f-03c069206b82","Type":"ContainerDied","Data":"569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682"} Jul 02 20:32:08.861923 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8" event={"ID":"704f746f-0999-4e43-9b7f-03c069206b82","Type":"ContainerDied","Data":"9424f40b6ca9ddaeb11d645519174c86722a5c91c82d1ed90f92d093b11b0982"} Jul 02 20:32:08.861923 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861671 2564 scope.go:117] "RemoveContainer" containerID="569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682" Jul 02 20:32:08.861923 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861662 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-model-cache" (OuterVolumeSpecName: "model-cache") pod "704f746f-0999-4e43-9b7f-03c069206b82" (UID: "704f746f-0999-4e43-9b7f-03c069206b82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.861923 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861551 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-home\") pod \"704f746f-0999-4e43-9b7f-03c069206b82\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " Jul 02 20:32:08.861923 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.861738 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f746f-0999-4e43-9b7f-03c069206b82-tls-certs\") pod \"704f746f-0999-4e43-9b7f-03c069206b82\" (UID: \"704f746f-0999-4e43-9b7f-03c069206b82\") " Jul 02 20:32:08.862174 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.862058 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.862174 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.862079 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.862174 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.862149 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-home" (OuterVolumeSpecName: "home") pod "704f746f-0999-4e43-9b7f-03c069206b82" (UID: "704f746f-0999-4e43-9b7f-03c069206b82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.863862 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.863839 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fe39959-7658-4a17-a52d-45401a4cb657" containerID="c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317" exitCode=0 Jul 02 20:32:08.863995 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.863876 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704f746f-0999-4e43-9b7f-03c069206b82-kube-api-access-kg4p6" (OuterVolumeSpecName: "kube-api-access-kg4p6") pod "704f746f-0999-4e43-9b7f-03c069206b82" (UID: "704f746f-0999-4e43-9b7f-03c069206b82"). InnerVolumeSpecName "kube-api-access-kg4p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:32:08.863995 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.863917 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" Jul 02 20:32:08.863995 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.863927 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" event={"ID":"7fe39959-7658-4a17-a52d-45401a4cb657","Type":"ContainerDied","Data":"c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317"} Jul 02 20:32:08.863995 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.863932 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-dshm" (OuterVolumeSpecName: "dshm") pod "704f746f-0999-4e43-9b7f-03c069206b82" (UID: "704f746f-0999-4e43-9b7f-03c069206b82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:32:08.863995 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.863957 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66" event={"ID":"7fe39959-7658-4a17-a52d-45401a4cb657","Type":"ContainerDied","Data":"9ea57a4ec942d0c61f10b6e18b490bcc7d00136569d1b87e61d3b61baa275f17"} Jul 02 20:32:08.864340 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.864218 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704f746f-0999-4e43-9b7f-03c069206b82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "704f746f-0999-4e43-9b7f-03c069206b82" (UID: "704f746f-0999-4e43-9b7f-03c069206b82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:32:08.878027 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.878006 2564 scope.go:117] "RemoveContainer" containerID="569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682" Jul 02 20:32:08.878315 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:32:08.878297 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682\": container with ID starting with 569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682 not found: ID does not exist" containerID="569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682" Jul 02 20:32:08.878364 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.878323 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682"} err="failed to get container status \"569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682\": rpc error: code = NotFound desc = could not find container \"569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682\": container with ID starting with 569aa70fb5cf36410ce677c52dcdcf6534e7825cf0091134a90efc2ba5483682 not found: ID does not exist" Jul 02 20:32:08.878364 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.878340 2564 scope.go:117] "RemoveContainer" containerID="c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317" Jul 02 20:32:08.886384 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.886366 2564 scope.go:117] "RemoveContainer" containerID="c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317" Jul 02 20:32:08.886629 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:32:08.886612 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317\": container with ID starting with c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317 not found: ID does not exist" containerID="c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317" Jul 02 20:32:08.886698 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.886639 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317"} err="failed to get container status \"c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317\": rpc error: code = NotFound desc = could not find container \"c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317\": container with ID starting with c2377312cd6b0901b0c0d00fcc8a44e465b701db33af76ba6294921e1222a317 not found: ID does not exist" Jul 02 20:32:08.890267 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.890246 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66"] Jul 02 20:32:08.894401 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.894383 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cd8cf5/sub-test-model-cd8cf5-kserve-74b6bd5cc7-9xf66"] Jul 02 20:32:08.963142 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.963106 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.963142 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.963142 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/704f746f-0999-4e43-9b7f-03c069206b82-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.963142 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.963152 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/704f746f-0999-4e43-9b7f-03c069206b82-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:08.963555 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:08.963185 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg4p6\" (UniqueName: \"kubernetes.io/projected/704f746f-0999-4e43-9b7f-03c069206b82-kube-api-access-kg4p6\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:32:09.054854 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:09.054822 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe39959-7658-4a17-a52d-45401a4cb657" path="/var/lib/kubelet/pods/7fe39959-7658-4a17-a52d-45401a4cb657/volumes" Jul 02 20:32:09.176526 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:09.176498 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8"] Jul 02 20:32:09.180396 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:09.180372 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/sub-test-model-fa5d1d-kserve-74b56769-bvdj8"] Jul 02 20:32:11.054265 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:11.054230 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704f746f-0999-4e43-9b7f-03c069206b82" path="/var/lib/kubelet/pods/704f746f-0999-4e43-9b7f-03c069206b82/volumes" Jul 02 20:32:13.319446 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:13.319410 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:32:36.493105 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.493028 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2"] Jul 02 20:32:36.493568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.493373 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704f746f-0999-4e43-9b7f-03c069206b82" containerName="main" Jul 02 20:32:36.493568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.493384 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="704f746f-0999-4e43-9b7f-03c069206b82" containerName="main" Jul 02 20:32:36.493568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.493417 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe39959-7658-4a17-a52d-45401a4cb657" containerName="main" Jul 02 20:32:36.493568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.493424 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe39959-7658-4a17-a52d-45401a4cb657" containerName="main" Jul 02 20:32:36.493568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.493526 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="704f746f-0999-4e43-9b7f-03c069206b82" containerName="main" Jul 02 20:32:36.493568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.493535 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe39959-7658-4a17-a52d-45401a4cb657" containerName="main" Jul 02 20:32:36.496797 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.496776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.500905 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.500885 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"kube-root-ca.crt\"" Jul 02 20:32:36.500905 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.500893 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"rate-test-model-fa5d1d-kserve-self-signed-certs\"" Jul 02 20:32:36.501096 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.500945 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"openshift-service-ca.crt\"" Jul 02 20:32:36.501096 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.501010 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa5d1d\"/\"default-dockercfg-hmmfs\"" Jul 02 20:32:36.514132 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.514105 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2"] Jul 02 20:32:36.607316 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.607276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-home\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.607316 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.607317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-tmp-dir\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.607568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.607406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c00653-c5e0-4908-994e-ebca5051d3fd-tls-certs\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.607568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.607428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-model-cache\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.607568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.607446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-dshm\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.607568 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.607525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgbw\" (UniqueName: \"kubernetes.io/projected/a1c00653-c5e0-4908-994e-ebca5051d3fd-kube-api-access-bzgbw\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708110 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c00653-c5e0-4908-994e-ebca5051d3fd-tls-certs\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708281 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-model-cache\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708281 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-dshm\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708281 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgbw\" (UniqueName: \"kubernetes.io/projected/a1c00653-c5e0-4908-994e-ebca5051d3fd-kube-api-access-bzgbw\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708281 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-home\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708281 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-tmp-dir\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708605 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708551 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-model-cache\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708662 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708631 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-tmp-dir\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.708755 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.708729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-home\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.710665 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.710640 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-dshm\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.710810 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.710792 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c00653-c5e0-4908-994e-ebca5051d3fd-tls-certs\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.723720 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.723693 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgbw\" (UniqueName: \"kubernetes.io/projected/a1c00653-c5e0-4908-994e-ebca5051d3fd-kube-api-access-bzgbw\") pod \"rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.805995 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.805915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:36.945181 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:36.945144 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2"] Jul 02 20:32:36.948134 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:32:36.948098 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c00653_c5e0_4908_994e_ebca5051d3fd.slice/crio-3ef9c88daa09680b539379f5c8a7e5657117ee526e8b97f9fbe24b6c4fa0d81e WatchSource:0}: Error finding container 3ef9c88daa09680b539379f5c8a7e5657117ee526e8b97f9fbe24b6c4fa0d81e: Status 404 returned error can't find the container with id 3ef9c88daa09680b539379f5c8a7e5657117ee526e8b97f9fbe24b6c4fa0d81e Jul 02 20:32:37.004690 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:37.004663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" event={"ID":"a1c00653-c5e0-4908-994e-ebca5051d3fd","Type":"ContainerStarted","Data":"3ef9c88daa09680b539379f5c8a7e5657117ee526e8b97f9fbe24b6c4fa0d81e"} Jul 02 20:32:38.010737 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:38.010702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" event={"ID":"a1c00653-c5e0-4908-994e-ebca5051d3fd","Type":"ContainerStarted","Data":"8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250"} Jul 02 20:32:38.075043 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:38.074986 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" podStartSLOduration=2.074970014 podStartE2EDuration="2.074970014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:32:38.072150203 +0000 UTC m=+2337.440886929" watchObservedRunningTime="2026-07-02 20:32:38.074970014 +0000 UTC m=+2337.443706730" Jul 02 20:32:40.312542 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:40.312504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:32:46.806194 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:46.806154 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:46.806194 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:46.806197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:46.818720 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:46.818693 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:32:47.064398 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:32:47.064318 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:33:04.211293 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:04.211254 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:33:16.804891 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:16.804856 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:33:23.801538 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:23.801504 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2"] Jul 02 20:33:25.669425 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:25.669393 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr"] Jul 02 20:33:25.674386 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:25.674362 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cd8cf5-29717070-7l9vr"] Jul 02 20:33:27.055292 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:27.055252 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2fb8c8a-b798-42d3-976b-624ab5b65a66" path="/var/lib/kubelet/pods/e2fb8c8a-b798-42d3-976b-624ab5b65a66/volumes" Jul 02 20:33:29.702686 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:29.702648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:33:38.820908 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:38.820849 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" podUID="a1c00653-c5e0-4908-994e-ebca5051d3fd" containerName="main" containerID="cri-o://8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250" gracePeriod=45 Jul 02 20:33:39.072992 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.072931 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:33:39.149009 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.148977 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzgbw\" (UniqueName: \"kubernetes.io/projected/a1c00653-c5e0-4908-994e-ebca5051d3fd-kube-api-access-bzgbw\") pod \"a1c00653-c5e0-4908-994e-ebca5051d3fd\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " Jul 02 20:33:39.149188 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.149055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c00653-c5e0-4908-994e-ebca5051d3fd-tls-certs\") pod \"a1c00653-c5e0-4908-994e-ebca5051d3fd\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " Jul 02 20:33:39.149188 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.149096 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-tmp-dir\") pod \"a1c00653-c5e0-4908-994e-ebca5051d3fd\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " Jul 02 20:33:39.149188 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.149120 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-home\") pod \"a1c00653-c5e0-4908-994e-ebca5051d3fd\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " Jul 02 20:33:39.149188 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.149144 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-dshm\") pod \"a1c00653-c5e0-4908-994e-ebca5051d3fd\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " Jul 02 20:33:39.149407 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.149382 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1c00653-c5e0-4908-994e-ebca5051d3fd" (UID: "a1c00653-c5e0-4908-994e-ebca5051d3fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:33:39.149537 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.149518 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:33:39.149839 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.149804 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-home" (OuterVolumeSpecName: "home") pod "a1c00653-c5e0-4908-994e-ebca5051d3fd" (UID: "a1c00653-c5e0-4908-994e-ebca5051d3fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:33:39.151277 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.151253 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-dshm" (OuterVolumeSpecName: "dshm") pod "a1c00653-c5e0-4908-994e-ebca5051d3fd" (UID: "a1c00653-c5e0-4908-994e-ebca5051d3fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:33:39.151376 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.151295 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c00653-c5e0-4908-994e-ebca5051d3fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1c00653-c5e0-4908-994e-ebca5051d3fd" (UID: "a1c00653-c5e0-4908-994e-ebca5051d3fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:33:39.151376 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.151308 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c00653-c5e0-4908-994e-ebca5051d3fd-kube-api-access-bzgbw" (OuterVolumeSpecName: "kube-api-access-bzgbw") pod "a1c00653-c5e0-4908-994e-ebca5051d3fd" (UID: "a1c00653-c5e0-4908-994e-ebca5051d3fd"). InnerVolumeSpecName "kube-api-access-bzgbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:33:39.249804 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.249771 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-model-cache\") pod \"a1c00653-c5e0-4908-994e-ebca5051d3fd\" (UID: \"a1c00653-c5e0-4908-994e-ebca5051d3fd\") " Jul 02 20:33:39.250000 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.249988 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c00653-c5e0-4908-994e-ebca5051d3fd-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:33:39.250045 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.250002 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:33:39.250045 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.249998 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1c00653-c5e0-4908-994e-ebca5051d3fd" (UID: "a1c00653-c5e0-4908-994e-ebca5051d3fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:33:39.250045 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.250012 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:33:39.250045 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.250026 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzgbw\" (UniqueName: \"kubernetes.io/projected/a1c00653-c5e0-4908-994e-ebca5051d3fd-kube-api-access-bzgbw\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:33:39.302095 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.302059 2564 generic.go:356] "Generic (PLEG): container finished" podID="a1c00653-c5e0-4908-994e-ebca5051d3fd" containerID="8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250" exitCode=0 Jul 02 20:33:39.302241 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.302105 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" Jul 02 20:33:39.302241 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.302101 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" event={"ID":"a1c00653-c5e0-4908-994e-ebca5051d3fd","Type":"ContainerDied","Data":"8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250"} Jul 02 20:33:39.302241 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.302216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2" event={"ID":"a1c00653-c5e0-4908-994e-ebca5051d3fd","Type":"ContainerDied","Data":"3ef9c88daa09680b539379f5c8a7e5657117ee526e8b97f9fbe24b6c4fa0d81e"} Jul 02 20:33:39.302241 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.302233 2564 scope.go:117] "RemoveContainer" containerID="8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250" Jul 02 20:33:39.312067 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.311985 2564 scope.go:117] "RemoveContainer" containerID="8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250" Jul 02 20:33:39.312946 ip-10-0-133-137 kubenswrapper[2564]: E0702 20:33:39.312923 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250\": container with ID starting with 8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250 not found: ID does not exist" containerID="8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250" Jul 02 20:33:39.313048 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.312952 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250"} err="failed to get container status \"8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250\": rpc error: code = NotFound desc = could not find container \"8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250\": container with ID starting with 8d420b6287edc9aba655b4d18db74ce9c1e18d7651a92c638ef90b9dac5fc250 not found: ID does not exist" Jul 02 20:33:39.323548 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.323490 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2"] Jul 02 20:33:39.327189 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.327171 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fa5d1d/rate-test-model-fa5d1d-kserve-7d958b7b95-6cmq2"] Jul 02 20:33:39.351109 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:39.351077 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c00653-c5e0-4908-994e-ebca5051d3fd-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\"" Jul 02 20:33:41.054980 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:33:41.054946 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c00653-c5e0-4908-994e-ebca5051d3fd" path="/var/lib/kubelet/pods/a1c00653-c5e0-4908-994e-ebca5051d3fd/volumes" Jul 02 20:34:12.016444 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:12.016403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jq7cl"] Jul 02 20:34:20.710071 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:20.710033 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717055-29vsr_fe07c079-de9b-4d66-8442-789d2aa01c1c/cleanup/0.log" Jul 02 20:34:20.820364 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:20.820331 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717070-gs7x4_20fa5d84-5e2d-41b3-ba4d-7928c820ff24/cleanup/0.log" Jul 02 20:34:21.545547 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:21.545519 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-844d4f46fb-7zt4j_a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b/copy-manifests/0.log" Jul 02 20:34:21.552006 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:21.551981 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-844d4f46fb-7zt4j_a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b/manager/0.log" Jul 02 20:34:23.171112 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:23.171079 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-blhh7_2a092568-d472-4840-96ca-b3d179b5cded/manager/0.log" Jul 02 20:34:23.277220 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:23.277188 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-h5qxq_07bf8241-38ab-4e69-81f4-6b9a9a872762/kuadrant-console-plugin/0.log" Jul 02 20:34:23.646316 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:23.646285 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jq7cl_52e0e3be-3979-45bf-8700-b800a10576a0/limitador/0.log" Jul 02 20:34:24.474047 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:24.474018 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b994b5d7-n85jw_cd296c3b-667a-4360-9d8e-6df95d158041/kube-auth-proxy/0.log" Jul 02 20:34:25.622559 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:25.622527 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbp9knq_f9c2a195-f8fa-409d-ac08-5a47ff614b9a/main/0.log" Jul 02 20:34:25.854412 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:25.854379 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-rtpll_48f432d0-a6da-4689-a236-db9d481d29e8/main/0.log" Jul 02 20:34:29.997238 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:29.997196 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xgw55/must-gather-bvwxb"] Jul 02 20:34:29.997666 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:29.997572 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c00653-c5e0-4908-994e-ebca5051d3fd" containerName="main" Jul 02 20:34:29.997666 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:29.997585 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c00653-c5e0-4908-994e-ebca5051d3fd" containerName="main" Jul 02 20:34:29.997666 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:29.997645 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c00653-c5e0-4908-994e-ebca5051d3fd" containerName="main" Jul 02 20:34:30.000390 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.000367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.003616 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.003590 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xgw55\"/\"default-dockercfg-v2l27\"" Jul 02 20:34:30.003744 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.003626 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xgw55\"/\"openshift-service-ca.crt\"" Jul 02 20:34:30.004783 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.004762 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xgw55\"/\"kube-root-ca.crt\"" Jul 02 20:34:30.016938 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.016910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgw55/must-gather-bvwxb"] Jul 02 20:34:30.110387 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.110352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btck\" (UniqueName: \"kubernetes.io/projected/047ff002-7184-45ae-92db-03c80ed6ff22-kube-api-access-7btck\") pod \"must-gather-bvwxb\" (UID: \"047ff002-7184-45ae-92db-03c80ed6ff22\") " pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.110611 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.110430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/047ff002-7184-45ae-92db-03c80ed6ff22-must-gather-output\") pod \"must-gather-bvwxb\" (UID: \"047ff002-7184-45ae-92db-03c80ed6ff22\") " pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.211837 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.211780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btck\" (UniqueName: \"kubernetes.io/projected/047ff002-7184-45ae-92db-03c80ed6ff22-kube-api-access-7btck\") pod \"must-gather-bvwxb\" (UID: \"047ff002-7184-45ae-92db-03c80ed6ff22\") " pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.212051 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.211901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/047ff002-7184-45ae-92db-03c80ed6ff22-must-gather-output\") pod \"must-gather-bvwxb\" (UID: \"047ff002-7184-45ae-92db-03c80ed6ff22\") " pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.212241 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.212221 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/047ff002-7184-45ae-92db-03c80ed6ff22-must-gather-output\") pod \"must-gather-bvwxb\" (UID: \"047ff002-7184-45ae-92db-03c80ed6ff22\") " pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.223254 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.223223 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btck\" (UniqueName: \"kubernetes.io/projected/047ff002-7184-45ae-92db-03c80ed6ff22-kube-api-access-7btck\") pod \"must-gather-bvwxb\" (UID: \"047ff002-7184-45ae-92db-03c80ed6ff22\") " pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.310244 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.310156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgw55/must-gather-bvwxb" Jul 02 20:34:30.441171 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.441144 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgw55/must-gather-bvwxb"] Jul 02 20:34:30.443228 ip-10-0-133-137 kubenswrapper[2564]: W0702 20:34:30.443201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod047ff002_7184_45ae_92db_03c80ed6ff22.slice/crio-890bfd174d21f47f60b01989725e585b34f6017d15d30a3bdd5a969427031973 WatchSource:0}: Error finding container 890bfd174d21f47f60b01989725e585b34f6017d15d30a3bdd5a969427031973: Status 404 returned error can't find the container with id 890bfd174d21f47f60b01989725e585b34f6017d15d30a3bdd5a969427031973 Jul 02 20:34:30.445303 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.445288 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:34:30.535088 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:30.535050 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgw55/must-gather-bvwxb" event={"ID":"047ff002-7184-45ae-92db-03c80ed6ff22","Type":"ContainerStarted","Data":"890bfd174d21f47f60b01989725e585b34f6017d15d30a3bdd5a969427031973"} Jul 02 20:34:31.542731 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:31.542631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgw55/must-gather-bvwxb" event={"ID":"047ff002-7184-45ae-92db-03c80ed6ff22","Type":"ContainerStarted","Data":"83d5b952aa495d79331a376c15adcd0c8149733e95523c9a42e010cc17260bdb"} Jul 02 20:34:31.542731 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:31.542688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgw55/must-gather-bvwxb" event={"ID":"047ff002-7184-45ae-92db-03c80ed6ff22","Type":"ContainerStarted","Data":"8660002d59cc38816cf8fe26aa31ea1da63eacb1bfac0b7aaf97506f96f88a00"} Jul 02 20:34:31.559186 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:31.559132 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xgw55/must-gather-bvwxb" podStartSLOduration=1.711311678 podStartE2EDuration="2.559114326s" totalImagesPullingTime="847.802648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:31.557567693 +0000 UTC m=+2450.926304420" watchObservedRunningTime="2026-07-02 20:34:31.559114326 +0000 UTC m=+2450.927851050" Jul 02 20:34:32.840765 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:32.840734 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g7p4v_48751f64-539d-4c0f-a40f-84264b0b6720/global-pull-secret-syncer/0.log" Jul 02 20:34:33.060283 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:33.060257 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wdjp6_f75a85a2-4ff5-42ec-938b-e9f3e343d2ae/konnectivity-agent/0.log" Jul 02 20:34:33.107154 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:33.107078 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-137.ec2.internal_c581f67582f2b7493459f83b250c1488/haproxy/0.log" Jul 02 20:34:38.521021 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:38.520989 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-blhh7_2a092568-d472-4840-96ca-b3d179b5cded/manager/0.log" Jul 02 20:34:38.569747 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:38.569699 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-h5qxq_07bf8241-38ab-4e69-81f4-6b9a9a872762/kuadrant-console-plugin/0.log" Jul 02 20:34:38.823195 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:38.823116 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jq7cl_52e0e3be-3979-45bf-8700-b800a10576a0/limitador/0.log" Jul 02 20:34:40.834917 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:40.834819 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4q4s9_29116bb4-6730-41f5-9c78-7b7efe438cfe/kube-state-metrics/0.log" Jul 02 20:34:40.859515 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:40.859486 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4q4s9_29116bb4-6730-41f5-9c78-7b7efe438cfe/kube-rbac-proxy-main/0.log" Jul 02 20:34:40.882180 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:40.882152 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4q4s9_29116bb4-6730-41f5-9c78-7b7efe438cfe/kube-rbac-proxy-self/0.log" Jul 02 20:34:40.945859 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:40.945825 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-7lhrr_40fda62b-664c-4637-a42d-acbc0f56657d/monitoring-plugin/0.log" Jul 02 20:34:40.977863 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:40.977823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k57rk_bcec82c2-f134-4207-9604-bc21397bd910/node-exporter/0.log" Jul 02 20:34:41.001231 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.001207 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k57rk_bcec82c2-f134-4207-9604-bc21397bd910/kube-rbac-proxy/0.log" Jul 02 20:34:41.028659 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.028632 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k57rk_bcec82c2-f134-4207-9604-bc21397bd910/init-textfile/0.log" Jul 02 20:34:41.328180 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.328149 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b40f16e3-ff32-4617-85f5-e507c2011906/prometheus/0.log" Jul 02 20:34:41.355539 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.355506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b40f16e3-ff32-4617-85f5-e507c2011906/config-reloader/0.log" Jul 02 20:34:41.384085 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.384048 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b40f16e3-ff32-4617-85f5-e507c2011906/thanos-sidecar/0.log" Jul 02 20:34:41.413803 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.413774 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b40f16e3-ff32-4617-85f5-e507c2011906/kube-rbac-proxy-web/0.log" Jul 02 20:34:41.439405 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.439375 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b40f16e3-ff32-4617-85f5-e507c2011906/kube-rbac-proxy/0.log" Jul 02 20:34:41.474006 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.473974 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b40f16e3-ff32-4617-85f5-e507c2011906/kube-rbac-proxy-thanos/0.log" Jul 02 20:34:41.507600 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.507572 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b40f16e3-ff32-4617-85f5-e507c2011906/init-config-reloader/0.log" Jul 02 20:34:41.540984 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.540953 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-w4wqk_a9bffdf5-591b-4022-9d16-78a62be3a0ff/prometheus-operator/0.log" Jul 02 20:34:41.565191 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:41.565164 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-w4wqk_a9bffdf5-591b-4022-9d16-78a62be3a0ff/kube-rbac-proxy/0.log" Jul 02 20:34:42.920788 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:42.920748 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4"] Jul 02 20:34:42.924859 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:42.924800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:42.935863 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:42.935831 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4"] Jul 02 20:34:43.058415 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.058383 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-proc\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.058590 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.058422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-lib-modules\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.058590 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.058452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-sys\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.058590 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.058564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-podres\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.058720 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.058613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pn7\" (UniqueName: \"kubernetes.io/projected/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-kube-api-access-h9pn7\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.159159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-proc\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.159234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-lib-modules\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.159294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-sys\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.159354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-podres\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.159389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pn7\" (UniqueName: \"kubernetes.io/projected/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-kube-api-access-h9pn7\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.160093 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-proc\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.160412 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-lib-modules\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.160670 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-sys\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.161101 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.161061 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-podres\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.169841 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.169783 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pn7\" (UniqueName: \"kubernetes.io/projected/4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58-kube-api-access-h9pn7\") pod \"perf-node-gather-daemonset-jjwv4\" (UID: \"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58\") " pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.256685 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.256595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.432174 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.432135 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4"] Jul 02 20:34:43.634955 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.634910 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" event={"ID":"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58","Type":"ContainerStarted","Data":"5e72e3f7be8e4a8a7167775d3731a6b2a40bb2689dd5ea43d6dc5dd44069b607"} Jul 02 20:34:43.635148 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.634982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" event={"ID":"4ae5d041-bf27-4ab1-a1a1-aabf96b6bb58","Type":"ContainerStarted","Data":"134c0f0dd69caf1328ded0e980dc1858f3069bef840a5e058b19145fe8114e94"} Jul 02 20:34:43.635332 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.635310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:43.657922 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:43.657871 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" podStartSLOduration=1.657854366 podStartE2EDuration="1.657854366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:34:43.655783774 +0000 UTC m=+2463.024520502" watchObservedRunningTime="2026-07-02 20:34:43.657854366 +0000 UTC m=+2463.026591095" Jul 02 20:34:45.975106 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:45.975076 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6qq7_856c4e6a-55d7-414e-b534-a6d36dc652f3/dns/0.log" Jul 02 20:34:45.997348 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:45.997318 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w6qq7_856c4e6a-55d7-414e-b534-a6d36dc652f3/kube-rbac-proxy/0.log" Jul 02 20:34:46.106047 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:46.106009 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qw9rz_4960089f-b7f6-4daa-ae08-c3d3df7cfe52/dns-node-resolver/0.log" Jul 02 20:34:46.674935 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:46.674898 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68f6c7f4bf-pvrmp_6ed273e4-d733-4174-9881-e939528c79f2/registry/0.log" Jul 02 20:34:46.754833 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:46.754803 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mbbgp_1f313f82-0e16-4e1c-8888-727cf6ea662c/node-ca/0.log" Jul 02 20:34:48.004593 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:48.004565 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b994b5d7-n85jw_cd296c3b-667a-4360-9d8e-6df95d158041/kube-auth-proxy/0.log" Jul 02 20:34:48.917436 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:48.917400 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kcm5k_a1d7c432-ae9a-43e2-b4cb-93dca17f6a67/serve-healthcheck-canary/0.log" Jul 02 20:34:49.536624 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:49.536594 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzwm5_1d595560-99c7-447b-9685-9e432ee1fde4/kube-rbac-proxy/0.log" Jul 02 20:34:49.562325 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:49.562300 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzwm5_1d595560-99c7-447b-9685-9e432ee1fde4/exporter/0.log" Jul 02 20:34:49.591885 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:49.591852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzwm5_1d595560-99c7-447b-9685-9e432ee1fde4/extractor/0.log" Jul 02 20:34:49.653068 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:49.653039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xgw55/perf-node-gather-daemonset-jjwv4" Jul 02 20:34:52.202322 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:52.202297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717055-29vsr_fe07c079-de9b-4d66-8442-789d2aa01c1c/cleanup/0.log" Jul 02 20:34:52.231069 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:52.231034 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717070-gs7x4_20fa5d84-5e2d-41b3-ba4d-7928c820ff24/cleanup/0.log" Jul 02 20:34:52.660716 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:52.660626 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-844d4f46fb-7zt4j_a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b/manager/0.log" Jul 02 20:34:52.690371 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:52.690345 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-844d4f46fb-7zt4j_a0f30d1e-c8bc-4ee3-8d78-9e9bf8e1e80b/copy-manifests/0.log" Jul 02 20:34:53.471983 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:53.471955 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b994b5d7-n85jw_cd296c3b-667a-4360-9d8e-6df95d158041/kube-auth-proxy/0.log" Jul 02 20:34:54.076198 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:54.076166 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56c987df7b-7mmbt_78b5a514-d28c-48a6-a348-c380b9c16a03/manager/0.log" Jul 02 20:34:55.033613 ip-10-0-133-137 kubenswrapper[2564]: I0702 20:34:55.033574 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b994b5d7-n85jw_cd296c3b-667a-4360-9d8e-6df95d158041/kube-auth-proxy/0.log"