Jul 15 19:01:24.006925 ip-10-0-130-245 systemd[1]: Starting Kubernetes Kubelet... Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.581199 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.590958 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.590981 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.590985 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.590988 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.590992 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.590996 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.590999 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:01:24.646149 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591001 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591004 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591007 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591010 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591013 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591016 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591020 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591023 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591026 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591029 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591032 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591035 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591038 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591040 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591043 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591046 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591048 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591051 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591054 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591056 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:01:24.647114 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591059 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591062 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591066 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591068 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591071 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591082 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591086 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591089 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591092 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591095 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591098 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591101 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591104 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591107 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591110 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591113 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591116 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591119 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591122 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:01:24.647662 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591127 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591131 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591136 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591139 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591142 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591146 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591149 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591154 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591158 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591162 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591165 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591168 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591171 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591174 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591177 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591180 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591183 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591185 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591188 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591192 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:01:24.648284 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591195 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591200 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591203 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591206 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591210 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591213 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591216 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591219 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591222 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591224 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591227 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591230 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591233 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591236 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591239 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591241 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591244 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591248 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591251 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:01:24.648883 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591254 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591258 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591260 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591264 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591267 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591270 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591273 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591276 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591279 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591281 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591284 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591287 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591290 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.591293 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593208 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593217 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593221 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593224 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593227 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593231 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:01:24.649453 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593234 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593237 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593240 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593243 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593246 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593249 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593253 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593256 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593259 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593262 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593265 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593269 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593273 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593275 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593278 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593281 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593284 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593287 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593289 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593292 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:01:24.650039 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593295 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593298 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593302 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593305 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593308 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593311 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593314 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593317 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593320 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593322 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593325 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593328 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593331 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593334 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593336 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593341 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593345 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593348 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593351 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:01:24.650650 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593355 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593359 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593362 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593366 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593369 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593373 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593376 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593379 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593382 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593385 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593387 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593390 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593393 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593395 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593398 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593401 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593404 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593407 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593409 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593412 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:01:24.651249 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593415 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593419 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593423 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593426 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593430 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593433 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593436 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593439 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593444 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593447 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593451 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593456 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593459 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593462 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593465 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593468 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593471 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593474 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593481 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:01:24.651878 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593485 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593488 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593491 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593493 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593496 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593499 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593502 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593504 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593507 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593510 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593513 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593516 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593519 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593522 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593524 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.593998 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594004 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594008 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594011 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594015 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:01:24.652416 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594018 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594021 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594023 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594026 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594031 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594035 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594038 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594042 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594045 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594048 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594052 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594065 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594069 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594072 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594075 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594082 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594085 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594088 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594091 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:01:24.834421 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594094 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:01:24.682534 ip-10-0-130-245 systemd[1]: Started Kubernetes Kubelet. Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594097 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594100 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594103 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594106 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594108 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594111 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594114 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594117 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594120 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594122 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594125 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594132 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594135 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594138 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594141 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594144 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594147 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594149 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594152 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:01:24.879080 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594155 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594158 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594161 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594164 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594168 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594172 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594176 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594185 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594189 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594192 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594195 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594198 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594201 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594203 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594206 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594210 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594213 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594216 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594219 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:01:24.880092 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594221 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594224 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594227 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594230 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594233 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594236 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594238 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594241 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594244 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594247 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594250 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594252 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594255 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594258 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594260 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594264 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594266 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594270 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594273 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594282 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:01:24.881120 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594286 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594288 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594291 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594294 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594296 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594299 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594302 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594305 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594308 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594310 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594313 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594315 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594318 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594321 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594324 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.594327 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594411 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594420 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594436 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594441 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594446 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 19:01:24.881995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594449 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594454 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594459 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594462 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594465 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594469 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594472 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594476 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594481 2561 flags.go:64] FLAG: --cgroup-root="" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594484 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594491 2561 flags.go:64] FLAG: --client-ca-file="" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594495 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594499 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594506 2561 flags.go:64] FLAG: --cluster-domain="" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594509 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594513 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594516 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594520 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594524 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594527 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594530 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594534 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594537 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594541 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 19:01:24.882868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594544 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594547 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594551 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594555 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594559 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594562 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594565 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594568 2561 flags.go:64] FLAG: --enable-server="true" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594571 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594577 2561 flags.go:64] FLAG: --event-burst="100" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594580 2561 flags.go:64] FLAG: --event-qps="50" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594583 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594587 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594590 2561 flags.go:64] FLAG: --eviction-hard="" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594594 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594597 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594602 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594605 2561 flags.go:64] FLAG: --eviction-soft="" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594610 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594613 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594617 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594620 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594624 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594627 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594630 2561 flags.go:64] FLAG: --feature-gates="" Jul 15 19:01:24.883663 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594634 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594637 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594641 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594644 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594648 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594651 2561 flags.go:64] FLAG: --help="false" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594654 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-245.ec2.internal" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594657 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594660 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594663 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594667 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594670 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594673 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594676 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594694 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594698 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594701 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594705 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594708 2561 flags.go:64] FLAG: --kube-reserved="" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594711 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594714 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594717 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594720 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594725 2561 flags.go:64] FLAG: --lock-file="" Jul 15 19:01:24.884556 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594728 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594733 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594736 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594742 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594747 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594750 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594753 2561 flags.go:64] FLAG: --logging-format="text" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594757 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594760 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594763 2561 flags.go:64] FLAG: --manifest-url="" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594767 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594772 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594775 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594781 2561 flags.go:64] FLAG: --max-pods="110" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594784 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594788 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594791 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594794 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594797 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594801 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594804 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594813 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594816 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594820 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 19:01:24.885428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594823 2561 flags.go:64] FLAG: --pod-cidr="" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594826 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594834 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594836 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594840 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594843 2561 flags.go:64] FLAG: --port="10250" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594847 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594851 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cfc04629258aeebe" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594855 2561 flags.go:64] FLAG: --qos-reserved="" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594858 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594864 2561 flags.go:64] FLAG: --register-node="true" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594867 2561 flags.go:64] FLAG: --register-with-taints="" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594871 2561 flags.go:64] FLAG: --registry-burst="10" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594875 2561 flags.go:64] FLAG: --registry-qps="5" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594878 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594881 2561 flags.go:64] FLAG: --reserved-memory="" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594885 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594889 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594892 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594895 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594898 2561 flags.go:64] FLAG: --runonce="false" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594901 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594905 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594908 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594911 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 19:01:24.886646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594914 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594917 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594920 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594924 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594927 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594931 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594934 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594937 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594940 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594944 2561 flags.go:64] FLAG: --system-cgroups="" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594947 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594952 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594955 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594958 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594964 2561 flags.go:64] FLAG: --tls-min-version="" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594967 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594970 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594976 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594979 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594982 2561 flags.go:64] FLAG: --v="2" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594987 2561 flags.go:64] FLAG: --version="false" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594992 2561 flags.go:64] FLAG: --vmodule="" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.594996 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.595000 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595135 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:01:24.887499 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595139 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595142 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595145 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595148 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595152 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595155 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595159 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595162 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595165 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595168 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595171 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595174 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595176 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595181 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595185 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595189 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595192 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595196 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595199 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:01:25.118617 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595201 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595206 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595212 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595214 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595217 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595222 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595225 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595227 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595230 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595233 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595237 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595240 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595243 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595246 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595248 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595251 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595254 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595257 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595259 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595262 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:01:25.119343 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595265 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595268 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595271 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595273 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595276 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595279 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595281 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595284 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595287 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595290 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595293 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595296 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595298 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595301 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595305 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595308 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595311 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595315 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595319 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595321 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:01:25.120052 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595324 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595327 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595330 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595334 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595337 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595339 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595342 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595345 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595347 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595350 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595353 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595355 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595358 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595361 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595364 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595366 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595369 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595372 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595375 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595378 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:01:25.120759 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595381 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595383 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595386 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595389 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595392 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595394 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595398 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595401 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595404 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595409 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595412 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595415 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595417 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595421 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595423 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595427 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595431 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595435 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:01:25.121347 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.595437 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.596051 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.604029 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.604054 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.604076 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.604087 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604158 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604164 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604168 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604171 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604174 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604178 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604181 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604184 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604187 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:01:25.121937 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604191 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604194 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604197 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604200 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604202 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604205 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604208 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604211 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604214 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604217 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604220 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604225 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604229 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604232 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604235 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604238 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604241 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604243 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604246 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:01:25.122610 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604249 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604258 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604262 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604267 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604270 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604273 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604276 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604278 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604281 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604284 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604287 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604289 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604292 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604295 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604298 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604301 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604304 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604307 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604309 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604312 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:01:25.123280 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604315 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604318 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604321 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604324 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604327 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604330 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604332 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604335 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604338 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604341 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604345 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604349 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604352 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604355 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604364 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604369 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604372 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604375 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604377 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604380 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:01:25.123948 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604382 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604385 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604388 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604391 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604394 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604397 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604399 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604402 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604405 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604408 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604410 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604413 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604416 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604420 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604424 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604428 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604431 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604435 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604438 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604441 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:01:25.124616 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604444 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604447 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604450 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604453 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604456 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604459 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604462 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604473 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604476 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604479 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604482 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.604488 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604646 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604652 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604655 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:01:25.125269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604659 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604662 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604665 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604668 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604671 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604676 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604679 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604697 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604699 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604704 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604707 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604713 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604716 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604719 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604722 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604727 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604730 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604734 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604737 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:01:25.125910 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604740 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604744 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604747 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604750 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604753 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604757 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604768 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604772 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604775 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604778 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604781 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604784 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604787 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604789 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604792 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604795 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604798 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604801 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604804 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604807 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:01:25.126449 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604810 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604813 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604816 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604819 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604823 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604826 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604830 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604833 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604836 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604839 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604842 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604845 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604848 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604851 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604853 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604856 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604859 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604864 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604867 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604876 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:01:25.127146 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604879 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604881 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604884 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604887 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604889 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604892 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604895 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604898 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604901 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604904 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604906 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604909 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604912 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604915 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604918 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604920 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604927 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604929 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604932 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604935 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:01:25.127735 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604938 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604940 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604943 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604946 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604949 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604951 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604954 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604957 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604960 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604964 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604967 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604970 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604979 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604982 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604984 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604987 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:01:25.128344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:24.604990 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.604996 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.605167 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.607544 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.609238 2561 server.go:1019] "Starting client certificate rotation" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.609370 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.609416 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.620128 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.621390 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.633488 2561 log.go:25] "Validated CRI v1 runtime API" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.638745 2561 log.go:25] "Validated CRI v1 image API" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.639113 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:01:25.128872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.639745 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 19:01:25.129288 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.645731 2561 fs.go:135] Filesystem UUIDs: map[2cc2a2c8-f385-4a12-af0d-2c893a26f8f9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 88b3e5b4-5902-4876-883d-34fd061fc588:/dev/nvme0n1p3] Jul 15 19:01:25.129288 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.645749 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.652272 2561 manager.go:216] Machine: {Timestamp:2026-07-15 19:01:24.650519692 +0000 UTC m=+0.484021237 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105726 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9d29b2a28734856c14fe8113238b SystemUUID:ec2f9d29-b2a2-8734-856c-14fe8113238b BootID:54462ed2-9640-4a28-8aa2-7af3ec04d6f8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7c:00:ed:7d:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7c:00:ed:7d:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:b8:7c:25:5f:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.652403 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.652503 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.656110 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.656152 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-245.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.657614 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.657626 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.657634 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.658194 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.659720 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.662953 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663118 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663154 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663172 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663194 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663210 2561 kubelet.go:398] "Adding apiserver pod source" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663224 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 19:01:25.129479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663619 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.663636 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.666142 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.666816 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zk4bx" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668065 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668096 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668258 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668271 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668277 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668283 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668291 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668297 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668835 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668848 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668858 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668868 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668882 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.668896 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.669789 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.669803 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.673557 2561 server.go:1268] "Started kubelet" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.673648 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.673736 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.681809 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.682079 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.683572 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-245.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.683993 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.683992 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.684470 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.684728 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zk4bx" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.686300 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.688217 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-245.ec2.internal.18c28bc5f1ca689a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-245.ec2.internal,UID:ip-10-0-130-245.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-245.ec2.internal,},FirstTimestamp:2026-07-15 19:01:24.67353001 +0000 UTC m=+0.507031557,LastTimestamp:2026-07-15 19:01:24.67353001 +0000 UTC m=+0.507031557,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-245.ec2.internal,}" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.689627 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.689635 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.689774 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.690224 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.690245 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.690224 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.690421 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.690426 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 15 19:01:25.130369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.691559 2561 factory.go:55] Registering systemd factory Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.691578 2561 factory.go:223] Registration of the systemd container factory successfully Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.691815 2561 factory.go:153] Registering CRI-O factory Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.691832 2561 factory.go:223] Registration of the crio container factory successfully Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.691886 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.691907 2561 factory.go:103] Registering Raw factory Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.691920 2561 manager.go:1196] Started watching for new ooms in manager Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.693131 2561 manager.go:349] Starting recovery of all containers Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.697415 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.703842 2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-245.ec2.internal\" not found" node="ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.704482 2561 manager.go:354] Recovery completed Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.709322 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.711788 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.711828 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.711843 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.712660 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.712671 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.712693 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.712674 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.713510 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.713537 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.713548 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.714045 2561 policy_none.go:49] "None policy: Start" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.714063 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.714076 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.715264 2561 policy_none.go:47] "Start" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.753417 2561 manager.go:337] "Starting Device Plugin manager" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.753458 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.753485 2561 server.go:87] "Starting device plugin registration server" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.753833 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.753846 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.754049 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.754184 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.754193 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.754751 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.754804 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.856117 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.858052 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.858085 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.858097 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.858129 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.868662 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.868687 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-245.ec2.internal\": node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.887583 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.924526 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.925882 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.925930 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.925965 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.926016 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 19:01:25.131695 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:24.928622 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:24.989753 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.027929 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal"] Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.028039 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.029921 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.029953 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.029968 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.031403 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.031615 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.031646 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.032478 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.032499 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.032509 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.032763 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.032782 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.032797 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.034185 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.034210 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.035008 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.035038 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.035048 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.060728 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-245.ec2.internal\" not found" node="ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.065832 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-245.ec2.internal\" not found" node="ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.090057 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.092884 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a50c844ab85f5e439f79e8473bb4424-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"0a50c844ab85f5e439f79e8473bb4424\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.092912 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a50c844ab85f5e439f79e8473bb4424-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"0a50c844ab85f5e439f79e8473bb4424\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.133156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.092932 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0d945a25ac134216e2142b82a7b9dc1-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"a0d945a25ac134216e2142b82a7b9dc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.191117 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.193497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a50c844ab85f5e439f79e8473bb4424-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"0a50c844ab85f5e439f79e8473bb4424\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.193530 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a50c844ab85f5e439f79e8473bb4424-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"0a50c844ab85f5e439f79e8473bb4424\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.193584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0d945a25ac134216e2142b82a7b9dc1-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"a0d945a25ac134216e2142b82a7b9dc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.193645 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a50c844ab85f5e439f79e8473bb4424-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"0a50c844ab85f5e439f79e8473bb4424\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.193655 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a50c844ab85f5e439f79e8473bb4424-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal\" (UID: \"0a50c844ab85f5e439f79e8473bb4424\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.193649 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0d945a25ac134216e2142b82a7b9dc1-config\") pod \"kube-apiserver-proxy-ip-10-0-130-245.ec2.internal\" (UID: \"a0d945a25ac134216e2142b82a7b9dc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.292127 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.367233 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.368280 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" Jul 15 19:01:25.410138 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.392316 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.661959 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.493197 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.661959 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.597237 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.661959 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.608199 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 19:01:25.661959 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.608359 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:01:25.661959 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.608422 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:01:25.877519 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.689573 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 18:56:25 +0000 UTC" deadline="2028-03-15 16:49:44.031452506 +0000 UTC" Jul 15 19:01:25.877519 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:25.689604 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14613h48m18.341851698s" Jul 15 19:01:25.877519 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.698028 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:25.877519 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.798403 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:26.131065 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.899598 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:26.131065 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:25.999822 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:26.131065 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.100534 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:26.353306 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.200632 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-245.ec2.internal\" not found" Jul 15 19:01:26.353306 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.232008 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:01:26.353306 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.242987 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:01:26.353306 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.290772 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" Jul 15 19:01:26.353306 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.305106 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:01:26.353306 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.306428 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" Jul 15 19:01:26.353306 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.316684 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:01:26.582369 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:26.372487 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d945a25ac134216e2142b82a7b9dc1.slice/crio-270ccb27dd8d32a228bf0670f57131c478624a6f35c7891bd41c4198d453b8f8 WatchSource:0}: Error finding container 270ccb27dd8d32a228bf0670f57131c478624a6f35c7891bd41c4198d453b8f8: Status 404 returned error can't find the container with id 270ccb27dd8d32a228bf0670f57131c478624a6f35c7891bd41c4198d453b8f8 Jul 15 19:01:26.582369 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.378433 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.667568 2561 apiserver.go:52] "Watching apiserver" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.674204 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.678370 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kc4vn","kube-system/global-pull-secret-syncer-fxjc5","openshift-cluster-node-tuning-operator/tuned-cc5np","openshift-dns/node-resolver-fsdk5","openshift-image-registry/node-ca-r67v4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal","openshift-multus/multus-additional-cni-plugins-jxddn","openshift-multus/multus-knmqj","openshift-multus/network-metrics-daemon-skp8g","kube-system/konnectivity-agent-g2qkd","kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc","openshift-network-diagnostics/network-check-target-4hlkp","openshift-network-operator/iptables-alerter-nrt24"] Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.681117 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.682270 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.682341 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.683475 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.683860 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.683937 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.683946 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.683985 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.684007 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.684257 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m75z5\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.684762 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.685962 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.686032 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.686032 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x8tls\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.686180 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.686834 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.687075 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h66gz\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.687437 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.687592 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.689275 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.689277 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.689428 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j2pz7\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.689455 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.689487 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.689498 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.689985 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6hxs6\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.690210 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.690603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.755875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.690728 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.691370 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.691680 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.692046 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.692179 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.693303 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.693491 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dcldl\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.693743 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.694486 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.695342 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.697470 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.697503 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.697519 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.697548 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.697859 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5hg4v\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.697974 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.698235 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.698307 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.698342 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zbq2x\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.698594 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.700836 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.700958 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.701101 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v6bb8\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.701257 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.701919 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-host\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.701942 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cf4779c-c578-4711-9a9d-68384e8a797e-host-slash\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.701959 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b5441de-eae8-4251-883e-7dff97969342-tmp\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.701975 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-cni-bin\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.701999 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-ovn\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.757360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702079 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-cni-netd\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702131 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-lib-modules\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-var-lib-kubelet\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702204 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/707f404d-e473-4ea4-99e1-3bfb155dfd23-dbus\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702251 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702283 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702300 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysctl-d\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702316 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/707f404d-e473-4ea4-99e1-3bfb155dfd23-kubelet-config\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702331 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gnpq\" (UniqueName: \"kubernetes.io/projected/7f9c4b60-829c-47dc-ac71-3798775f5d55-kube-api-access-2gnpq\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntxp\" (UniqueName: \"kubernetes.io/projected/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-kube-api-access-qntxp\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702408 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-systemd-units\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702430 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702447 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-ovnkube-script-lib\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702480 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysconfig\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2cf4779c-c578-4711-9a9d-68384e8a797e-iptables-alerter-script\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.758401 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702517 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mq76\" (UniqueName: \"kubernetes.io/projected/2cf4779c-c578-4711-9a9d-68384e8a797e-kube-api-access-8mq76\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702538 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-ovnkube-config\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-sys\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-os-release\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702596 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-systemd\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b5441de-eae8-4251-883e-7dff97969342-etc-tuned\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702657 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f9c4b60-829c-47dc-ac71-3798775f5d55-host\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cnibin\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702798 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-log-socket\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21744bb-e296-49be-84f0-d6708c0f1459-tmp-dir\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-system-cni-dir\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-modprobe-d\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-run\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.702994 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-etc-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-node-log\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759156 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703029 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-run-ovn-kubernetes\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703055 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxhp\" (UniqueName: \"kubernetes.io/projected/e21744bb-e296-49be-84f0-d6708c0f1459-kube-api-access-rkxhp\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703371 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703414 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-kubernetes\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703453 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-run-netns\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703530 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703600 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703706 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7818af85-6523-469d-b917-243de65f815f-ovn-node-metrics-cert\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703753 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpqj\" (UniqueName: \"kubernetes.io/projected/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-kube-api-access-9dpqj\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703781 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f9c4b60-829c-47dc-ac71-3798775f5d55-serviceca\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703813 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-kubelet\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703853 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-systemd\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703884 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5z5\" (UniqueName: \"kubernetes.io/projected/1b5441de-eae8-4251-883e-7dff97969342-kube-api-access-9f5z5\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703919 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-slash\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.703967 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-env-overrides\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.704553 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e21744bb-e296-49be-84f0-d6708c0f1459-hosts-file\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.759913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.706135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysctl-conf\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.760526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.706384 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxh8k\" (UniqueName: \"kubernetes.io/projected/7818af85-6523-469d-b917-243de65f815f-kube-api-access-wxh8k\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.760526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.706402 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-var-lib-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.760526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.706424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.760526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.712828 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:01:26.760526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.737294 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ftmqq" Jul 15 19:01:26.760526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.745218 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:01:26.760526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.745291 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ftmqq" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.791061 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807467 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-run-ovn-kubernetes\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807505 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-socket-dir-parent\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807526 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxhp\" (UniqueName: \"kubernetes.io/projected/e21744bb-e296-49be-84f0-d6708c0f1459-kube-api-access-rkxhp\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807547 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807568 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-kubernetes\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-run-netns\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-run-ovn-kubernetes\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807651 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-run-netns\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807674 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-kubernetes\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807714 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-netns\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807751 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db0b5165-3965-45ac-8af1-9550ba236543-multus-daemon-config\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807780 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807807 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.807913 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7818af85-6523-469d-b917-243de65f815f-ovn-node-metrics-cert\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807943 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpqj\" (UniqueName: \"kubernetes.io/projected/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-kube-api-access-9dpqj\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.970214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807946 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.807965 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7838d9e7-4153-4796-ad6c-532b49b0d483-agent-certs\") pod \"konnectivity-agent-g2qkd\" (UID: \"7838d9e7-4153-4796-ad6c-532b49b0d483\") " pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.808012 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs podName:398e6ee4-ffb7-418d-a5f1-31b4528fcae7 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:27.307986471 +0000 UTC m=+3.141488009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs") pod "network-metrics-daemon-skp8g" (UID: "398e6ee4-ffb7-418d-a5f1-31b4528fcae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808034 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f9c4b60-829c-47dc-ac71-3798775f5d55-serviceca\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-kubelet\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808091 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-systemd\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808128 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808139 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-kubelet\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808146 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-hostroot\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808197 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-systemd\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-conf-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808243 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-multus-certs\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808270 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5z5\" (UniqueName: \"kubernetes.io/projected/1b5441de-eae8-4251-883e-7dff97969342-kube-api-access-9f5z5\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808274 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808287 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-slash\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808306 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-env-overrides\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808330 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tj9\" (UniqueName: \"kubernetes.io/projected/db0b5165-3965-45ac-8af1-9550ba236543-kube-api-access-t9tj9\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.971291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808349 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e21744bb-e296-49be-84f0-d6708c0f1459-hosts-file\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808350 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-slash\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808366 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysctl-conf\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808427 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxh8k\" (UniqueName: \"kubernetes.io/projected/7818af85-6523-469d-b917-243de65f815f-kube-api-access-wxh8k\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808435 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e21744bb-e296-49be-84f0-d6708c0f1459-hosts-file\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808460 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-cnibin\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-k8s-cni-cncf-io\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-socket-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808526 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f9c4b60-829c-47dc-ac71-3798775f5d55-serviceca\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808540 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-registration-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808569 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-var-lib-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808597 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808616 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-var-lib-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808636 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysctl-conf\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808637 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-kubelet\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808685 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk92m\" (UniqueName: \"kubernetes.io/projected/f68b906b-3e1f-4aea-b118-868d42d9582a-kube-api-access-xk92m\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808707 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-host\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.972241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808674 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cf4779c-c578-4711-9a9d-68384e8a797e-host-slash\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808742 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-cni-multus\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808760 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-etc-selinux\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808765 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-host\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808791 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b5441de-eae8-4251-883e-7dff97969342-tmp\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808808 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cf4779c-c578-4711-9a9d-68384e8a797e-host-slash\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808827 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-cni-bin\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808849 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808863 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-cni-bin\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-ovn\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-env-overrides\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-cni-netd\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808918 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-ovn\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808924 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-lib-modules\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808945 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-var-lib-kubelet\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808972 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/707f404d-e473-4ea4-99e1-3bfb155dfd23-dbus\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.972905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.808998 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809023 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809022 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-host-cni-netd\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809040 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-var-lib-kubelet\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809079 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysctl-d\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809113 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/707f404d-e473-4ea4-99e1-3bfb155dfd23-kubelet-config\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.809137 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809149 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/707f404d-e473-4ea4-99e1-3bfb155dfd23-dbus\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809140 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.809194 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret podName:707f404d-e473-4ea4-99e1-3bfb155dfd23 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:27.309172755 +0000 UTC m=+3.142674288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret") pod "global-pull-secret-syncer-fxjc5" (UID: "707f404d-e473-4ea4-99e1-3bfb155dfd23") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809240 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnpq\" (UniqueName: \"kubernetes.io/projected/7f9c4b60-829c-47dc-ac71-3798775f5d55-kube-api-access-2gnpq\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809271 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntxp\" (UniqueName: \"kubernetes.io/projected/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-kube-api-access-qntxp\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809281 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysctl-d\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809026 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-lib-modules\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809303 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-systemd-units\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809328 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-ovnkube-script-lib\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.973675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809387 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7838d9e7-4153-4796-ad6c-532b49b0d483-konnectivity-ca\") pod \"konnectivity-agent-g2qkd\" (UID: \"7838d9e7-4153-4796-ad6c-532b49b0d483\") " pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809403 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-systemd-units\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809414 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-system-cni-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809425 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/707f404d-e473-4ea4-99e1-3bfb155dfd23-kubelet-config\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809442 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-os-release\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809470 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-cni-bin\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809590 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysconfig\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809657 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2cf4779c-c578-4711-9a9d-68384e8a797e-iptables-alerter-script\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809690 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-sysconfig\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809736 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mq76\" (UniqueName: \"kubernetes.io/projected/2cf4779c-c578-4711-9a9d-68384e8a797e-kube-api-access-8mq76\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-ovnkube-config\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.809739 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-run-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-ovnkube-script-lib\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810148 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-cni-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810190 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-etc-kubernetes\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.974317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810215 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-device-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810239 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-sys-fs\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-sys\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810333 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-os-release\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-systemd\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2cf4779c-c578-4711-9a9d-68384e8a797e-iptables-alerter-script\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b5441de-eae8-4251-883e-7dff97969342-etc-tuned\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810386 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-sys\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810431 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f9c4b60-829c-47dc-ac71-3798775f5d55-host\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810460 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cnibin\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810469 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-systemd\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810471 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-os-release\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810490 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f9c4b60-829c-47dc-ac71-3798775f5d55-host\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810524 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-cnibin\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810519 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-log-socket\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810541 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7818af85-6523-469d-b917-243de65f815f-ovnkube-config\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810563 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db0b5165-3965-45ac-8af1-9550ba236543-cni-binary-copy\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810574 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-log-socket\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810594 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21744bb-e296-49be-84f0-d6708c0f1459-tmp-dir\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810629 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-system-cni-dir\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810709 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-system-cni-dir\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810806 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-modprobe-d\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810836 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-run\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-etc-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810887 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-node-log\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810915 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-etc-modprobe-d\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810946 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b5441de-eae8-4251-883e-7dff97969342-run\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.810974 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-node-log\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.811007 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7818af85-6523-469d-b917-243de65f815f-etc-openvswitch\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.811007 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21744bb-e296-49be-84f0-d6708c0f1459-tmp-dir\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.812412 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b5441de-eae8-4251-883e-7dff97969342-tmp\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.817244 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b5441de-eae8-4251-883e-7dff97969342-etc-tuned\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.817764 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7818af85-6523-469d-b917-243de65f815f-ovn-node-metrics-cert\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.820501 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxhp\" (UniqueName: \"kubernetes.io/projected/e21744bb-e296-49be-84f0-d6708c0f1459-kube-api-access-rkxhp\") pod \"node-resolver-fsdk5\" (UID: \"e21744bb-e296-49be-84f0-d6708c0f1459\") " pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.821479 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:26.975670 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.821503 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.821520 2561 projected.go:196] Error preparing data for projected volume kube-api-access-4ppq4 for pod openshift-network-diagnostics/network-check-target-4hlkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:26.821614 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4 podName:cd2b941e-8f72-40a0-bdd7-c50333aebbc4 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:27.321573561 +0000 UTC m=+3.155075114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4ppq4" (UniqueName: "kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4") pod "network-check-target-4hlkp" (UID: "cd2b941e-8f72-40a0-bdd7-c50333aebbc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.822560 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntxp\" (UniqueName: \"kubernetes.io/projected/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-kube-api-access-qntxp\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.822807 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mq76\" (UniqueName: \"kubernetes.io/projected/2cf4779c-c578-4711-9a9d-68384e8a797e-kube-api-access-8mq76\") pod \"iptables-alerter-nrt24\" (UID: \"2cf4779c-c578-4711-9a9d-68384e8a797e\") " pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.823199 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnpq\" (UniqueName: \"kubernetes.io/projected/7f9c4b60-829c-47dc-ac71-3798775f5d55-kube-api-access-2gnpq\") pod \"node-ca-r67v4\" (UID: \"7f9c4b60-829c-47dc-ac71-3798775f5d55\") " pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.823255 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxh8k\" (UniqueName: \"kubernetes.io/projected/7818af85-6523-469d-b917-243de65f815f-kube-api-access-wxh8k\") pod \"ovnkube-node-kc4vn\" (UID: \"7818af85-6523-469d-b917-243de65f815f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.823358 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpqj\" (UniqueName: \"kubernetes.io/projected/bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a-kube-api-access-9dpqj\") pod \"multus-additional-cni-plugins-jxddn\" (UID: \"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a\") " pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.823930 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5z5\" (UniqueName: \"kubernetes.io/projected/1b5441de-eae8-4251-883e-7dff97969342-kube-api-access-9f5z5\") pod \"tuned-cc5np\" (UID: \"1b5441de-eae8-4251-883e-7dff97969342\") " pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:26.894523 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a50c844ab85f5e439f79e8473bb4424.slice/crio-bb5751ec8a6b77285244fe1ca59d8d12edcb9a8a95b231cc21e91d9f0fdc60c7 WatchSource:0}: Error finding container bb5751ec8a6b77285244fe1ca59d8d12edcb9a8a95b231cc21e91d9f0fdc60c7: Status 404 returned error can't find the container with id bb5751ec8a6b77285244fe1ca59d8d12edcb9a8a95b231cc21e91d9f0fdc60c7 Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911152 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-cni-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911183 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-etc-kubernetes\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-device-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-sys-fs\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-device-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db0b5165-3965-45ac-8af1-9550ba236543-cni-binary-copy\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.976438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911305 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-cni-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911320 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-socket-dir-parent\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911339 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-sys-fs\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-netns\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911311 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-etc-kubernetes\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911365 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-socket-dir-parent\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db0b5165-3965-45ac-8af1-9550ba236543-multus-daemon-config\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911411 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-netns\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7838d9e7-4153-4796-ad6c-532b49b0d483-agent-certs\") pod \"konnectivity-agent-g2qkd\" (UID: \"7838d9e7-4153-4796-ad6c-532b49b0d483\") " pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-hostroot\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911500 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-conf-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911518 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-multus-certs\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911547 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-hostroot\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911580 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-multus-conf-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911731 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tj9\" (UniqueName: \"kubernetes.io/projected/db0b5165-3965-45ac-8af1-9550ba236543-kube-api-access-t9tj9\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911765 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-cnibin\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-k8s-cni-cncf-io\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911808 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-socket-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-registration-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-kubelet\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911862 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk92m\" (UniqueName: \"kubernetes.io/projected/f68b906b-3e1f-4aea-b118-868d42d9582a-kube-api-access-xk92m\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911880 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-cni-multus\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911946 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-registration-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911958 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db0b5165-3965-45ac-8af1-9550ba236543-cni-binary-copy\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911968 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/db0b5165-3965-45ac-8af1-9550ba236543-multus-daemon-config\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.911966 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-multus-certs\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912006 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-cni-multus\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-run-k8s-cni-cncf-io\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912029 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-socket-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912050 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-etc-selinux\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912070 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-kubelet\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912081 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912126 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912150 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7838d9e7-4153-4796-ad6c-532b49b0d483-konnectivity-ca\") pod \"konnectivity-agent-g2qkd\" (UID: \"7838d9e7-4153-4796-ad6c-532b49b0d483\") " pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912192 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-system-cni-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912194 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-cnibin\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.977743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912217 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-os-release\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912229 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f68b906b-3e1f-4aea-b118-868d42d9582a-etc-selinux\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912247 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-cni-bin\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912272 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-os-release\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912290 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-system-cni-dir\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912307 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db0b5165-3965-45ac-8af1-9550ba236543-host-var-lib-cni-bin\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.912602 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7838d9e7-4153-4796-ad6c-532b49b0d483-konnectivity-ca\") pod \"konnectivity-agent-g2qkd\" (UID: \"7838d9e7-4153-4796-ad6c-532b49b0d483\") " pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.914064 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7838d9e7-4153-4796-ad6c-532b49b0d483-agent-certs\") pod \"konnectivity-agent-g2qkd\" (UID: \"7838d9e7-4153-4796-ad6c-532b49b0d483\") " pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.919185 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.924418 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tj9\" (UniqueName: \"kubernetes.io/projected/db0b5165-3965-45ac-8af1-9550ba236543-kube-api-access-t9tj9\") pod \"multus-knmqj\" (UID: \"db0b5165-3965-45ac-8af1-9550ba236543\") " pod="openshift-multus/multus-knmqj" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.924432 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk92m\" (UniqueName: \"kubernetes.io/projected/f68b906b-3e1f-4aea-b118-868d42d9582a-kube-api-access-xk92m\") pod \"aws-ebs-csi-driver-node-g92sc\" (UID: \"f68b906b-3e1f-4aea-b118-868d42d9582a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.930511 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"0a50c844ab85f5e439f79e8473bb4424","Type":"ContainerStarted","Data":"bb5751ec8a6b77285244fe1ca59d8d12edcb9a8a95b231cc21e91d9f0fdc60c7"} Jul 15 19:01:26.978343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.931546 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" event={"ID":"a0d945a25ac134216e2142b82a7b9dc1","Type":"ContainerStarted","Data":"270ccb27dd8d32a228bf0670f57131c478624a6f35c7891bd41c4198d453b8f8"} Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.996186 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jxddn" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:26.999160 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cc5np" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.008218 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc1ee78_8528_41f4_ac5c_0fa034ab2b9a.slice/crio-26c1ae4f8102a76f74d326c2d3d005941cdc75b581f34ecc9b729ab4b19e98f0 WatchSource:0}: Error finding container 26c1ae4f8102a76f74d326c2d3d005941cdc75b581f34ecc9b729ab4b19e98f0: Status 404 returned error can't find the container with id 26c1ae4f8102a76f74d326c2d3d005941cdc75b581f34ecc9b729ab4b19e98f0 Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.008859 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5441de_eae8_4251_883e_7dff97969342.slice/crio-3b922f9e23c1e2a537962a318d2679d73bff58b0120f65a0da278cf1414711e5 WatchSource:0}: Error finding container 3b922f9e23c1e2a537962a318d2679d73bff58b0120f65a0da278cf1414711e5: Status 404 returned error can't find the container with id 3b922f9e23c1e2a537962a318d2679d73bff58b0120f65a0da278cf1414711e5 Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.009176 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fsdk5" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.014483 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r67v4" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.018870 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21744bb_e296_49be_84f0_d6708c0f1459.slice/crio-968304559e02ec968feecd4084226ebe0900ae480b3de5dc493e546dcb969d5b WatchSource:0}: Error finding container 968304559e02ec968feecd4084226ebe0900ae480b3de5dc493e546dcb969d5b: Status 404 returned error can't find the container with id 968304559e02ec968feecd4084226ebe0900ae480b3de5dc493e546dcb969d5b Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.026262 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nrt24" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.030703 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9c4b60_829c_47dc_ac71_3798775f5d55.slice/crio-936f5809e7c016034895834a13db1158e7b17474aa9dd8c5385f6725b8beee7e WatchSource:0}: Error finding container 936f5809e7c016034895834a13db1158e7b17474aa9dd8c5385f6725b8beee7e: Status 404 returned error can't find the container with id 936f5809e7c016034895834a13db1158e7b17474aa9dd8c5385f6725b8beee7e Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.030781 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.034893 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf4779c_c578_4711_9a9d_68384e8a797e.slice/crio-8e239a9fdfa3332ebddd2a3362084486599cfb33781645737a11821a62dc7a7d WatchSource:0}: Error finding container 8e239a9fdfa3332ebddd2a3362084486599cfb33781645737a11821a62dc7a7d: Status 404 returned error can't find the container with id 8e239a9fdfa3332ebddd2a3362084486599cfb33781645737a11821a62dc7a7d Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.035449 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.039512 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7818af85_6523_469d_b917_243de65f815f.slice/crio-7376a705040e502fc3fa40c934712addac6f9e10a6307007fe31f81eda6b7b46 WatchSource:0}: Error finding container 7376a705040e502fc3fa40c934712addac6f9e10a6307007fe31f81eda6b7b46: Status 404 returned error can't find the container with id 7376a705040e502fc3fa40c934712addac6f9e10a6307007fe31f81eda6b7b46 Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.041015 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knmqj" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.045149 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7838d9e7_4153_4796_ad6c_532b49b0d483.slice/crio-bfd16b474a58e1136a02b2981ef7ac90bd5f6e7f4dc3652b4b95ad544bc2dd3e WatchSource:0}: Error finding container bfd16b474a58e1136a02b2981ef7ac90bd5f6e7f4dc3652b4b95ad544bc2dd3e: Status 404 returned error can't find the container with id bfd16b474a58e1136a02b2981ef7ac90bd5f6e7f4dc3652b4b95ad544bc2dd3e Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.046191 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" Jul 15 19:01:27.172157 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:01:27.056507 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68b906b_3e1f_4aea_b118_868d42d9582a.slice/crio-aedffdec4c64bae99befa2764dfc352e3b395fb1cbe877c2ad892a2d022d8240 WatchSource:0}: Error finding container aedffdec4c64bae99befa2764dfc352e3b395fb1cbe877c2ad892a2d022d8240: Status 404 returned error can't find the container with id aedffdec4c64bae99befa2764dfc352e3b395fb1cbe877c2ad892a2d022d8240 Jul 15 19:01:27.406250 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.315129 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:27.406250 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.315206 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:27.406250 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.315314 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:27.406250 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.315318 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:27.406250 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.315374 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs podName:398e6ee4-ffb7-418d-a5f1-31b4528fcae7 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:28.315357447 +0000 UTC m=+4.148859001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs") pod "network-metrics-daemon-skp8g" (UID: "398e6ee4-ffb7-418d-a5f1-31b4528fcae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:27.406250 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.315390 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret podName:707f404d-e473-4ea4-99e1-3bfb155dfd23 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:28.315383035 +0000 UTC m=+4.148884567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret") pod "global-pull-secret-syncer-fxjc5" (UID: "707f404d-e473-4ea4-99e1-3bfb155dfd23") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:27.517638 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.416237 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:27.517638 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.416446 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:27.517638 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.416477 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:27.517638 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.416492 2561 projected.go:196] Error preparing data for projected volume kube-api-access-4ppq4 for pod openshift-network-diagnostics/network-check-target-4hlkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:27.517638 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.416563 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4 podName:cd2b941e-8f72-40a0-bdd7-c50333aebbc4 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:28.416542822 +0000 UTC m=+4.250044369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4ppq4" (UniqueName: "kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4") pod "network-check-target-4hlkp" (UID: "cd2b941e-8f72-40a0-bdd7-c50333aebbc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:27.748013 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.747845 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 18:56:27 +0000 UTC" deadline="2028-04-09 03:53:40.953211122 +0000 UTC" Jul 15 19:01:27.748013 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.747894 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15200h52m13.205322321s" Jul 15 19:01:27.927126 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.926510 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:27.927126 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:27.926666 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:27.941879 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.941780 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2qkd" event={"ID":"7838d9e7-4153-4796-ad6c-532b49b0d483","Type":"ContainerStarted","Data":"bfd16b474a58e1136a02b2981ef7ac90bd5f6e7f4dc3652b4b95ad544bc2dd3e"} Jul 15 19:01:27.948462 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.948327 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nrt24" event={"ID":"2cf4779c-c578-4711-9a9d-68384e8a797e","Type":"ContainerStarted","Data":"8e239a9fdfa3332ebddd2a3362084486599cfb33781645737a11821a62dc7a7d"} Jul 15 19:01:27.950945 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.950860 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fsdk5" event={"ID":"e21744bb-e296-49be-84f0-d6708c0f1459","Type":"ContainerStarted","Data":"968304559e02ec968feecd4084226ebe0900ae480b3de5dc493e546dcb969d5b"} Jul 15 19:01:27.959052 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.958923 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knmqj" event={"ID":"db0b5165-3965-45ac-8af1-9550ba236543","Type":"ContainerStarted","Data":"a3b2a4d4dc963322b433022ed003806d8ca933006a0a7eae3f246e36e341a0b7"} Jul 15 19:01:27.969589 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.969444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"7376a705040e502fc3fa40c934712addac6f9e10a6307007fe31f81eda6b7b46"} Jul 15 19:01:27.973547 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.973409 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r67v4" event={"ID":"7f9c4b60-829c-47dc-ac71-3798775f5d55","Type":"ContainerStarted","Data":"936f5809e7c016034895834a13db1158e7b17474aa9dd8c5385f6725b8beee7e"} Jul 15 19:01:27.977064 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.977027 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cc5np" event={"ID":"1b5441de-eae8-4251-883e-7dff97969342","Type":"ContainerStarted","Data":"3b922f9e23c1e2a537962a318d2679d73bff58b0120f65a0da278cf1414711e5"} Jul 15 19:01:27.982493 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.982455 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerStarted","Data":"26c1ae4f8102a76f74d326c2d3d005941cdc75b581f34ecc9b729ab4b19e98f0"} Jul 15 19:01:27.986598 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:27.986566 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" event={"ID":"f68b906b-3e1f-4aea-b118-868d42d9582a","Type":"ContainerStarted","Data":"aedffdec4c64bae99befa2764dfc352e3b395fb1cbe877c2ad892a2d022d8240"} Jul 15 19:01:28.326268 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:28.325776 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:28.326268 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:28.325927 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:28.326268 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.325975 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:28.326268 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.326059 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret podName:707f404d-e473-4ea4-99e1-3bfb155dfd23 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:30.326036483 +0000 UTC m=+6.159538025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret") pod "global-pull-secret-syncer-fxjc5" (UID: "707f404d-e473-4ea4-99e1-3bfb155dfd23") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:28.326268 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.326068 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:28.326268 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.326140 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs podName:398e6ee4-ffb7-418d-a5f1-31b4528fcae7 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:30.326106434 +0000 UTC m=+6.159607972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs") pod "network-metrics-daemon-skp8g" (UID: "398e6ee4-ffb7-418d-a5f1-31b4528fcae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:28.426896 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:28.426857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:28.427747 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.427086 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:28.427747 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.427111 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:28.427747 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.427124 2561 projected.go:196] Error preparing data for projected volume kube-api-access-4ppq4 for pod openshift-network-diagnostics/network-check-target-4hlkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:28.427747 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.427185 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4 podName:cd2b941e-8f72-40a0-bdd7-c50333aebbc4 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:30.427166644 +0000 UTC m=+6.260668192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4ppq4" (UniqueName: "kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4") pod "network-check-target-4hlkp" (UID: "cd2b941e-8f72-40a0-bdd7-c50333aebbc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:28.749085 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:28.748873 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 18:56:27 +0000 UTC" deadline="2028-01-05 05:31:51.388778862 +0000 UTC" Jul 15 19:01:28.749085 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:28.748943 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12922h30m22.639839464s" Jul 15 19:01:28.928712 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:28.927043 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:28.928712 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.927193 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:28.928712 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:28.927663 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:28.928712 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:28.927806 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:29.926425 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:29.926391 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:29.926632 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:29.926529 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:30.345351 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:30.344566 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:30.345351 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:30.344677 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:30.345351 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.344823 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:30.345351 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.344897 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs podName:398e6ee4-ffb7-418d-a5f1-31b4528fcae7 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:34.344873568 +0000 UTC m=+10.178375116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs") pod "network-metrics-daemon-skp8g" (UID: "398e6ee4-ffb7-418d-a5f1-31b4528fcae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:30.345351 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.344824 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:30.345351 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.344983 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret podName:707f404d-e473-4ea4-99e1-3bfb155dfd23 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:34.34496107 +0000 UTC m=+10.178462622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret") pod "global-pull-secret-syncer-fxjc5" (UID: "707f404d-e473-4ea4-99e1-3bfb155dfd23") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:30.445361 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:30.445251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:30.445547 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.445426 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:30.445547 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.445456 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:30.445547 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.445472 2561 projected.go:196] Error preparing data for projected volume kube-api-access-4ppq4 for pod openshift-network-diagnostics/network-check-target-4hlkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:30.445547 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.445540 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4 podName:cd2b941e-8f72-40a0-bdd7-c50333aebbc4 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:34.445519299 +0000 UTC m=+10.279020853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4ppq4" (UniqueName: "kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4") pod "network-check-target-4hlkp" (UID: "cd2b941e-8f72-40a0-bdd7-c50333aebbc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:30.927726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:30.926312 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:30.927726 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.926454 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:30.927726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:30.927172 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:30.927726 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:30.927324 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:31.927222 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:31.927181 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:31.927752 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:31.927333 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:32.927273 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:32.926634 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:32.927273 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:32.926764 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:32.927273 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:32.927127 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:32.927273 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:32.927232 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:33.927171 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:33.927115 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:33.927403 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:33.927275 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:34.383879 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:34.383206 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:34.383879 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:34.383292 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:34.383879 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.383444 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:34.383879 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.383505 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret podName:707f404d-e473-4ea4-99e1-3bfb155dfd23 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:42.383487011 +0000 UTC m=+18.216988556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret") pod "global-pull-secret-syncer-fxjc5" (UID: "707f404d-e473-4ea4-99e1-3bfb155dfd23") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:34.383879 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.383833 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:34.384317 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.383896 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs podName:398e6ee4-ffb7-418d-a5f1-31b4528fcae7 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:42.383879491 +0000 UTC m=+18.217381024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs") pod "network-metrics-daemon-skp8g" (UID: "398e6ee4-ffb7-418d-a5f1-31b4528fcae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:34.485227 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:34.484558 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:34.485227 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.484766 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:34.485227 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.484787 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:34.485227 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.484800 2561 projected.go:196] Error preparing data for projected volume kube-api-access-4ppq4 for pod openshift-network-diagnostics/network-check-target-4hlkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:34.485227 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.484857 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4 podName:cd2b941e-8f72-40a0-bdd7-c50333aebbc4 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:42.484839394 +0000 UTC m=+18.318340927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4ppq4" (UniqueName: "kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4") pod "network-check-target-4hlkp" (UID: "cd2b941e-8f72-40a0-bdd7-c50333aebbc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:34.928064 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:34.927797 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:34.928550 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:34.927903 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:34.928550 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.928162 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:34.928550 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:34.928292 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:35.927153 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:35.926635 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:35.927153 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:35.926766 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:36.928749 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:36.928719 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:36.929153 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:36.928724 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:36.929153 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:36.928857 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:36.929153 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:36.928926 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:37.926818 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:37.926780 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:37.926972 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:37.926895 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:38.926564 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:38.926527 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:38.927052 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:38.926527 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:38.927052 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:38.926681 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:38.927052 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:38.926729 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:39.926761 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:39.926717 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:39.927155 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:39.926851 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:40.926491 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:40.926459 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:40.926685 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:40.926502 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:40.926685 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:40.926593 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:40.926786 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:40.926716 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:41.926486 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:41.926439 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:41.926749 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:41.926568 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:42.450675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:42.450638 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:42.451167 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:42.450704 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:42.451167 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.450815 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:42.451167 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.450823 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:42.451167 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.450870 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret podName:707f404d-e473-4ea4-99e1-3bfb155dfd23 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:58.450856642 +0000 UTC m=+34.284358175 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret") pod "global-pull-secret-syncer-fxjc5" (UID: "707f404d-e473-4ea4-99e1-3bfb155dfd23") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:42.451167 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.450890 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs podName:398e6ee4-ffb7-418d-a5f1-31b4528fcae7 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:58.450875382 +0000 UTC m=+34.284376912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs") pod "network-metrics-daemon-skp8g" (UID: "398e6ee4-ffb7-418d-a5f1-31b4528fcae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:42.551788 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:42.551747 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:42.552042 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.551926 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:42.552042 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.551948 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:42.552042 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.551958 2561 projected.go:196] Error preparing data for projected volume kube-api-access-4ppq4 for pod openshift-network-diagnostics/network-check-target-4hlkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:42.552042 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.552016 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4 podName:cd2b941e-8f72-40a0-bdd7-c50333aebbc4 nodeName:}" failed. No retries permitted until 2026-07-15 19:01:58.551997195 +0000 UTC m=+34.385498745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4ppq4" (UniqueName: "kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4") pod "network-check-target-4hlkp" (UID: "cd2b941e-8f72-40a0-bdd7-c50333aebbc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:42.926951 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:42.926861 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:42.927118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:42.926861 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:42.927118 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.927007 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:42.927118 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:42.927108 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:43.926452 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:43.926413 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:43.926910 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:43.926587 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:44.928618 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:44.928558 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:44.929072 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:44.928760 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:44.929334 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:44.929298 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:44.929465 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:44.929444 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:45.926209 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:45.926173 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:45.926410 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:45.926297 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:46.926832 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:46.926761 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:46.927396 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:46.926761 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:46.927396 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:46.926911 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:46.927396 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:46.926982 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:47.929095 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:47.926760 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:47.929095 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:47.927107 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:48.036536 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.036500 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" event={"ID":"a0d945a25ac134216e2142b82a7b9dc1","Type":"ContainerStarted","Data":"a3f424e53e9e8b28770b859a5028858b554ba2795dd67880ac61e87448d2d799"} Jul 15 19:01:48.037923 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.037850 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knmqj" event={"ID":"db0b5165-3965-45ac-8af1-9550ba236543","Type":"ContainerStarted","Data":"5ee98983f71b6644078ea655e6d26baa5717d358fbb56e83a41762aa6a0d0dd2"} Jul 15 19:01:48.039702 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.039671 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:01:48.040095 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.040072 2561 generic.go:356] "Generic (PLEG): container finished" podID="7818af85-6523-469d-b917-243de65f815f" containerID="6f5d41b6c0f0052a2aae447560ac879a4fea6008711e13045ce5028413b348ff" exitCode=1 Jul 15 19:01:48.040203 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.040129 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"2277a694f1ae1a5923c1d2701379538ab8504d92621f9e29d4eb19f81c63db9d"} Jul 15 19:01:48.040203 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.040181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerDied","Data":"6f5d41b6c0f0052a2aae447560ac879a4fea6008711e13045ce5028413b348ff"} Jul 15 19:01:48.040289 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.040213 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"85b388b34be99eb3f5f2ee671a2c836587d3b9a7d466921d4cc9d36d03b03c39"} Jul 15 19:01:48.041749 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.041727 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r67v4" event={"ID":"7f9c4b60-829c-47dc-ac71-3798775f5d55","Type":"ContainerStarted","Data":"cc6948e6eb926449553b5f6591ef5268ec79ead553dc994703286fed59019352"} Jul 15 19:01:48.043041 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.043023 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cc5np" event={"ID":"1b5441de-eae8-4251-883e-7dff97969342","Type":"ContainerStarted","Data":"52ca61916bf4a35bfddd06830e4d21618b220048170c1a9ffbf59fd9bd09cc55"} Jul 15 19:01:48.044450 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.044431 2561 generic.go:356] "Generic (PLEG): container finished" podID="bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a" containerID="2d50315e6610e8046a835a2d218ed00d5d4ad23e9fa170d12b43ff96eedfc8af" exitCode=0 Jul 15 19:01:48.044540 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.044491 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerDied","Data":"2d50315e6610e8046a835a2d218ed00d5d4ad23e9fa170d12b43ff96eedfc8af"} Jul 15 19:01:48.046275 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.046251 2561 generic.go:356] "Generic (PLEG): container finished" podID="0a50c844ab85f5e439f79e8473bb4424" containerID="7a79622d759da7f9fbfb14824f4c43507a202b6ac9175b021bd0851e95245970" exitCode=0 Jul 15 19:01:48.046351 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.046284 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"0a50c844ab85f5e439f79e8473bb4424","Type":"ContainerDied","Data":"7a79622d759da7f9fbfb14824f4c43507a202b6ac9175b021bd0851e95245970"} Jul 15 19:01:48.047831 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.047805 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" event={"ID":"f68b906b-3e1f-4aea-b118-868d42d9582a","Type":"ContainerStarted","Data":"989383f5587292b3bebd26f61f75b47f197d0dc5150398771c7ae8d58b5ccbe2"} Jul 15 19:01:48.049324 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.049299 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2qkd" event={"ID":"7838d9e7-4153-4796-ad6c-532b49b0d483","Type":"ContainerStarted","Data":"da65d915b254fdfc37ba9e71ec092b26a5756e0f525de98b1241995aed1f6028"} Jul 15 19:01:48.050948 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.050843 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-245.ec2.internal" podStartSLOduration=22.050826221 podStartE2EDuration="22.050826221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:48.049642106 +0000 UTC m=+23.883143654" watchObservedRunningTime="2026-07-15 19:01:48.050826221 +0000 UTC m=+23.884327777" Jul 15 19:01:48.051063 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.051020 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fsdk5" event={"ID":"e21744bb-e296-49be-84f0-d6708c0f1459","Type":"ContainerStarted","Data":"5641220095da5f96ee28a3b1e701072d0f9ff153be7ea31a262b5f38b8e8f6c7"} Jul 15 19:01:48.064029 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.063857 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g2qkd" podStartSLOduration=2.595675523 podStartE2EDuration="23.06384193s" totalImagesPullingTime="20.468166407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:48.06372038 +0000 UTC m=+23.897221954" watchObservedRunningTime="2026-07-15 19:01:48.06384193 +0000 UTC m=+23.897343482" Jul 15 19:01:48.080849 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.080808 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-knmqj" podStartSLOduration=2.558930255 podStartE2EDuration="23.080792589s" totalImagesPullingTime="20.521862334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:48.080488691 +0000 UTC m=+23.913990256" watchObservedRunningTime="2026-07-15 19:01:48.080792589 +0000 UTC m=+23.914294140" Jul 15 19:01:48.137668 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.137624 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r67v4" podStartSLOduration=7.418488104 podStartE2EDuration="23.137592926s" totalImagesPullingTime="15.719104822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:48.114731115 +0000 UTC m=+23.948232687" watchObservedRunningTime="2026-07-15 19:01:48.137592926 +0000 UTC m=+23.971094469" Jul 15 19:01:48.138213 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.138193 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cc5np" podStartSLOduration=2.635487673 podStartE2EDuration="23.13818661s" totalImagesPullingTime="20.502698937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:48.137589724 +0000 UTC m=+23.971091304" watchObservedRunningTime="2026-07-15 19:01:48.13818661 +0000 UTC m=+23.971688162" Jul 15 19:01:48.151736 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.151685 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fsdk5" podStartSLOduration=7.089727977 podStartE2EDuration="23.151667822s" totalImagesPullingTime="16.061939845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:48.151254474 +0000 UTC m=+23.984756007" watchObservedRunningTime="2026-07-15 19:01:48.151667822 +0000 UTC m=+23.985169374" Jul 15 19:01:48.926477 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.926438 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:48.926678 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:48.926571 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:48.926758 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:48.926695 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:48.926847 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:48.926812 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:49.057143 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.057117 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:01:49.057895 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.057518 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"5d3f44e00bd63f5308e7eb7d638ea9cabcfbd171f0a3b7d7d7beaa596bc1ad75"} Jul 15 19:01:49.057895 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.057596 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"8615cf5c8c37fe8f98367361838a8044a141c43112d3de554cf5eeb9cd4872e6"} Jul 15 19:01:49.057895 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.057649 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"1c4ac60c89bb299ab25f6c6d2b4f94cc423039d95d499b015ef2e9ce575e93bd"} Jul 15 19:01:49.059378 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.059350 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nrt24" event={"ID":"2cf4779c-c578-4711-9a9d-68384e8a797e","Type":"ContainerStarted","Data":"f50053f6806af57f01072a13c861430c3080337cb82c285b1058c0b4016e6191"} Jul 15 19:01:49.076440 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.076388 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nrt24" podStartSLOduration=8.362167438 podStartE2EDuration="24.076367058s" totalImagesPullingTime="15.71419962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:49.075527207 +0000 UTC m=+24.909028777" watchObservedRunningTime="2026-07-15 19:01:49.076367058 +0000 UTC m=+24.909868611" Jul 15 19:01:49.133874 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.133824 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:49.134546 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.134526 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:49.342019 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.341994 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 19:01:49.769933 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.769840 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T19:01:49.342014812Z","UUID":"d5158ae2-9d3e-4272-bb32-c773573b6fad","Handler":null,"Name":"","Endpoint":""} Jul 15 19:01:49.771501 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.771477 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 19:01:49.771660 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.771523 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 19:01:49.926176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:49.926137 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:49.926413 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:49.926280 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:50.063738 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:50.063700 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" event={"ID":"0a50c844ab85f5e439f79e8473bb4424","Type":"ContainerStarted","Data":"86741f74e6ca5eae38befdbd24dd5042da2594013f3339373837caed2125fc02"} Jul 15 19:01:50.065819 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:50.065786 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" event={"ID":"f68b906b-3e1f-4aea-b118-868d42d9582a","Type":"ContainerStarted","Data":"89980c274d1a75f3cc3af7b03b6d7c00ec9750d59efc5b17ab4d108ee8b64f5a"} Jul 15 19:01:50.066110 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:50.066090 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:50.066718 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:50.066697 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g2qkd" Jul 15 19:01:50.081725 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:50.081670 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-245.ec2.internal" podStartSLOduration=24.081648436 podStartE2EDuration="24.081648436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:50.081177542 +0000 UTC m=+25.914679095" watchObservedRunningTime="2026-07-15 19:01:50.081648436 +0000 UTC m=+25.915149990" Jul 15 19:01:50.926660 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:50.926390 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:50.926865 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:50.926436 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:50.926865 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:50.926792 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:50.926865 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:50.926846 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:51.070398 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:51.070367 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" event={"ID":"f68b906b-3e1f-4aea-b118-868d42d9582a","Type":"ContainerStarted","Data":"07321976b8586df9d58d1a04d334f3ed46756aa083037b56f32306682a4ff3d7"} Jul 15 19:01:51.074064 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:51.074040 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:01:51.074468 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:51.074433 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"23e51620f0b44b84d0777da10c1659e5b990be070461bf09f9ca8620b2c2ed29"} Jul 15 19:01:51.090447 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:51.090387 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g92sc" podStartSLOduration=3.275851182 podStartE2EDuration="26.0903705s" totalImagesPullingTime="22.814519318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:51.089675936 +0000 UTC m=+26.923177516" watchObservedRunningTime="2026-07-15 19:01:51.0903705 +0000 UTC m=+26.923872051" Jul 15 19:01:51.927027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:51.926993 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:51.927259 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:51.927122 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:52.927085 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:52.926735 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:52.927085 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:52.926735 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:52.927974 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:52.927142 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:52.927974 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:52.927280 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:53.085676 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.085650 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:01:53.086031 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.086007 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"310fa309ed2f37719e34b380c9f1611dbaa3c39dfff58f95674b2265b6792062"} Jul 15 19:01:53.086305 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.086283 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:53.086477 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.086455 2561 scope.go:117] "RemoveContainer" containerID="6f5d41b6c0f0052a2aae447560ac879a4fea6008711e13045ce5028413b348ff" Jul 15 19:01:53.088073 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.088051 2561 generic.go:356] "Generic (PLEG): container finished" podID="bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a" containerID="b2c6c723bf193d06ffb15ec8e6a9997626321e27c66c1459a6678b256b5761c8" exitCode=0 Jul 15 19:01:53.088184 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.088096 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerDied","Data":"b2c6c723bf193d06ffb15ec8e6a9997626321e27c66c1459a6678b256b5761c8"} Jul 15 19:01:53.104104 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.104078 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:53.927055 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:53.927018 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:53.927308 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:53.927161 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:54.094288 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.094265 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:01:54.094660 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.094634 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" event={"ID":"7818af85-6523-469d-b917-243de65f815f","Type":"ContainerStarted","Data":"81890784c852b58d9835c6cae9c7f2e794d0707015ec5eba01b2083017d1eb88"} Jul 15 19:01:54.094855 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.094836 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:54.094953 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.094896 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:54.110576 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.110550 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:01:54.189399 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.189304 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" podStartSLOduration=8.657415288 podStartE2EDuration="29.189288722s" totalImagesPullingTime="20.531873434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:01:54.155813665 +0000 UTC m=+29.989315215" watchObservedRunningTime="2026-07-15 19:01:54.189288722 +0000 UTC m=+30.022790275" Jul 15 19:01:54.833814 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.833783 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxjc5"] Jul 15 19:01:54.834059 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.833935 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:54.834059 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:54.834028 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:54.837445 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.837401 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-skp8g"] Jul 15 19:01:54.837588 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.837550 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:54.837707 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:54.837683 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:54.837986 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.837964 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4hlkp"] Jul 15 19:01:54.838077 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:54.838055 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:54.838168 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:54.838147 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:55.098867 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:55.098776 2561 generic.go:356] "Generic (PLEG): container finished" podID="bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a" containerID="a6d7a5000ec3b497732f0519f557c07abf099264e9fc0062fe5ae9198615b7dc" exitCode=0 Jul 15 19:01:55.098867 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:55.098833 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerDied","Data":"a6d7a5000ec3b497732f0519f557c07abf099264e9fc0062fe5ae9198615b7dc"} Jul 15 19:01:56.926670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:56.926453 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:56.927172 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:56.926457 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:56.927172 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:56.926752 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:56.927172 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:56.926522 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:56.927172 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:56.926813 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:01:56.927172 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:56.926901 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:57.106668 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:57.106634 2561 generic.go:356] "Generic (PLEG): container finished" podID="bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a" containerID="a84b7f73671ed183467b7d42dd8fe4c5eb88e262c113de65ceb3d0cfb128bfc4" exitCode=0 Jul 15 19:01:57.106838 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:57.106676 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerDied","Data":"a84b7f73671ed183467b7d42dd8fe4c5eb88e262c113de65ceb3d0cfb128bfc4"} Jul 15 19:01:58.469302 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:58.469269 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:58.469921 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:58.469343 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:58.469921 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.469423 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:58.469921 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.469439 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:58.469921 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.469486 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret podName:707f404d-e473-4ea4-99e1-3bfb155dfd23 nodeName:}" failed. No retries permitted until 2026-07-15 19:02:30.469470601 +0000 UTC m=+66.302972134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret") pod "global-pull-secret-syncer-fxjc5" (UID: "707f404d-e473-4ea4-99e1-3bfb155dfd23") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:01:58.469921 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.469502 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs podName:398e6ee4-ffb7-418d-a5f1-31b4528fcae7 nodeName:}" failed. No retries permitted until 2026-07-15 19:02:30.469496098 +0000 UTC m=+66.302997631 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs") pod "network-metrics-daemon-skp8g" (UID: "398e6ee4-ffb7-418d-a5f1-31b4528fcae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:01:58.569872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:58.569841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:58.570045 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.570016 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:01:58.570045 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.570044 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:01:58.570162 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.570057 2561 projected.go:196] Error preparing data for projected volume kube-api-access-4ppq4 for pod openshift-network-diagnostics/network-check-target-4hlkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:58.570162 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.570122 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4 podName:cd2b941e-8f72-40a0-bdd7-c50333aebbc4 nodeName:}" failed. No retries permitted until 2026-07-15 19:02:30.57010323 +0000 UTC m=+66.403604763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4ppq4" (UniqueName: "kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4") pod "network-check-target-4hlkp" (UID: "cd2b941e-8f72-40a0-bdd7-c50333aebbc4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:01:58.926454 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:58.926420 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:01:58.926652 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:58.926460 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:01:58.926652 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:01:58.926424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:01:58.926652 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.926562 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4hlkp" podUID="cd2b941e-8f72-40a0-bdd7-c50333aebbc4" Jul 15 19:01:58.926874 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.926652 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxjc5" podUID="707f404d-e473-4ea4-99e1-3bfb155dfd23" Jul 15 19:01:58.926874 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:01:58.926750 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-skp8g" podUID="398e6ee4-ffb7-418d-a5f1-31b4528fcae7" Jul 15 19:02:00.468443 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.468360 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-245.ec2.internal" event="NodeReady" Jul 15 19:02:00.468979 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.468515 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 19:02:00.524468 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.524432 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8fdhd"] Jul 15 19:02:00.549387 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.549343 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nvscg"] Jul 15 19:02:00.549575 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.549488 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.553189 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.553162 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgrg9\"" Jul 15 19:02:00.553315 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.553299 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 19:02:00.553531 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.553512 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 19:02:00.573534 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.573498 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fdhd"] Jul 15 19:02:00.573732 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.573551 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvscg"] Jul 15 19:02:00.573732 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.573668 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.577787 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.577752 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jgfdg\"" Jul 15 19:02:00.578063 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.578046 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 19:02:00.578168 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.578121 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 19:02:00.581035 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.581007 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 19:02:00.620891 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.620855 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7vtpv"] Jul 15 19:02:00.641510 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.641474 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vtpv"] Jul 15 19:02:00.641691 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.641630 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.646856 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.646710 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 19:02:00.647007 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.646885 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 19:02:00.647110 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.647094 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 19:02:00.647184 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.647168 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 19:02:00.647291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.647273 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tn2w9\"" Jul 15 19:02:00.684215 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.684179 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a08c025-91a2-4c90-9f14-9bc9683083d8-tmp-dir\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.684416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.684224 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bce3637-bc3e-46ff-8456-030ce270bc5b-cert\") pod \"ingress-canary-nvscg\" (UID: \"0bce3637-bc3e-46ff-8456-030ce270bc5b\") " pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.684416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.684250 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a08c025-91a2-4c90-9f14-9bc9683083d8-config-volume\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.684416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.684310 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a08c025-91a2-4c90-9f14-9bc9683083d8-metrics-tls\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.684416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.684352 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkbp\" (UniqueName: \"kubernetes.io/projected/0bce3637-bc3e-46ff-8456-030ce270bc5b-kube-api-access-jvkbp\") pod \"ingress-canary-nvscg\" (UID: \"0bce3637-bc3e-46ff-8456-030ce270bc5b\") " pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.684416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.684377 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp6tz\" (UniqueName: \"kubernetes.io/projected/9a08c025-91a2-4c90-9f14-9bc9683083d8-kube-api-access-wp6tz\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.785481 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785447 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59771a4c-e433-42eb-82c4-86e0af23ba57-crio-socket\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.785713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785494 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp6tz\" (UniqueName: \"kubernetes.io/projected/9a08c025-91a2-4c90-9f14-9bc9683083d8-kube-api-access-wp6tz\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.785713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85r7l\" (UniqueName: \"kubernetes.io/projected/59771a4c-e433-42eb-82c4-86e0af23ba57-kube-api-access-85r7l\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.785713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785570 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a08c025-91a2-4c90-9f14-9bc9683083d8-tmp-dir\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.785713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785592 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59771a4c-e433-42eb-82c4-86e0af23ba57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.785713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bce3637-bc3e-46ff-8456-030ce270bc5b-cert\") pod \"ingress-canary-nvscg\" (UID: \"0bce3637-bc3e-46ff-8456-030ce270bc5b\") " pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.785713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59771a4c-e433-42eb-82c4-86e0af23ba57-data-volume\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.786014 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785716 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a08c025-91a2-4c90-9f14-9bc9683083d8-config-volume\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.786014 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785733 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59771a4c-e433-42eb-82c4-86e0af23ba57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.786014 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785757 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a08c025-91a2-4c90-9f14-9bc9683083d8-metrics-tls\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.786014 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.785793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkbp\" (UniqueName: \"kubernetes.io/projected/0bce3637-bc3e-46ff-8456-030ce270bc5b-kube-api-access-jvkbp\") pod \"ingress-canary-nvscg\" (UID: \"0bce3637-bc3e-46ff-8456-030ce270bc5b\") " pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.786014 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.786003 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a08c025-91a2-4c90-9f14-9bc9683083d8-tmp-dir\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.786364 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.786335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a08c025-91a2-4c90-9f14-9bc9683083d8-config-volume\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.790098 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.790072 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a08c025-91a2-4c90-9f14-9bc9683083d8-metrics-tls\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.790220 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.790174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bce3637-bc3e-46ff-8456-030ce270bc5b-cert\") pod \"ingress-canary-nvscg\" (UID: \"0bce3637-bc3e-46ff-8456-030ce270bc5b\") " pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.794375 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.794350 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkbp\" (UniqueName: \"kubernetes.io/projected/0bce3637-bc3e-46ff-8456-030ce270bc5b-kube-api-access-jvkbp\") pod \"ingress-canary-nvscg\" (UID: \"0bce3637-bc3e-46ff-8456-030ce270bc5b\") " pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.794375 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.794372 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp6tz\" (UniqueName: \"kubernetes.io/projected/9a08c025-91a2-4c90-9f14-9bc9683083d8-kube-api-access-wp6tz\") pod \"dns-default-8fdhd\" (UID: \"9a08c025-91a2-4c90-9f14-9bc9683083d8\") " pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.860469 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.860427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:00.884571 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.884522 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvscg" Jul 15 19:02:00.886058 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886037 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59771a4c-e433-42eb-82c4-86e0af23ba57-crio-socket\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.886202 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85r7l\" (UniqueName: \"kubernetes.io/projected/59771a4c-e433-42eb-82c4-86e0af23ba57-kube-api-access-85r7l\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.886202 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886143 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59771a4c-e433-42eb-82c4-86e0af23ba57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.886202 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59771a4c-e433-42eb-82c4-86e0af23ba57-data-volume\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.886202 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886203 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59771a4c-e433-42eb-82c4-86e0af23ba57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.886488 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886308 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59771a4c-e433-42eb-82c4-86e0af23ba57-crio-socket\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.886637 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59771a4c-e433-42eb-82c4-86e0af23ba57-data-volume\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.886862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.886835 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59771a4c-e433-42eb-82c4-86e0af23ba57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.888888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.888866 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59771a4c-e433-42eb-82c4-86e0af23ba57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.894497 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.894474 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85r7l\" (UniqueName: \"kubernetes.io/projected/59771a4c-e433-42eb-82c4-86e0af23ba57-kube-api-access-85r7l\") pod \"insights-runtime-extractor-7vtpv\" (UID: \"59771a4c-e433-42eb-82c4-86e0af23ba57\") " pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:00.926672 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.926636 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:02:00.926855 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.926637 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:02:00.926855 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.926637 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:02:00.929666 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.929472 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:02:00.929666 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.929492 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:02:00.929666 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.929515 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7hbwd\"" Jul 15 19:02:00.929666 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.929472 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:02:00.929965 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.929874 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:02:00.929965 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.929908 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j7rjq\"" Jul 15 19:02:00.952415 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:00.952392 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vtpv" Jul 15 19:02:01.313644 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.313352 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b74cdd84-dn5v9"] Jul 15 19:02:01.336599 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.336565 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b74cdd84-dn5v9"] Jul 15 19:02:01.336761 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.336685 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.339519 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.339494 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 19:02:01.339668 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.339494 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 19:02:01.339792 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.339775 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5mhbd\"" Jul 15 19:02:01.339846 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.339786 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 19:02:01.339846 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.339813 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 19:02:01.339952 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.339878 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 19:02:01.339952 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.339910 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 19:02:01.340115 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.340100 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 19:02:01.492705 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.492497 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-oauth-serving-cert\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.492705 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.492574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-oauth-config\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.492705 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.492603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-kube-api-access-pbqhx\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.492705 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.492666 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-service-ca\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.492705 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.492697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-serving-cert\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.493343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.492746 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-config\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.493425 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.493403 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvscg"] Jul 15 19:02:01.496352 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.496269 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fdhd"] Jul 15 19:02:01.497062 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.497043 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vtpv"] Jul 15 19:02:01.515744 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:01.515710 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bce3637_bc3e_46ff_8456_030ce270bc5b.slice/crio-db6021ab624f686d9b227f30abc36664a242554cae932f05811aa023963ed906 WatchSource:0}: Error finding container db6021ab624f686d9b227f30abc36664a242554cae932f05811aa023963ed906: Status 404 returned error can't find the container with id db6021ab624f686d9b227f30abc36664a242554cae932f05811aa023963ed906 Jul 15 19:02:01.516122 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:01.516082 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a08c025_91a2_4c90_9f14_9bc9683083d8.slice/crio-07989928eb2e908e7ae4790935d11b63484af793a642b2e0faba11547c7834b3 WatchSource:0}: Error finding container 07989928eb2e908e7ae4790935d11b63484af793a642b2e0faba11547c7834b3: Status 404 returned error can't find the container with id 07989928eb2e908e7ae4790935d11b63484af793a642b2e0faba11547c7834b3 Jul 15 19:02:01.517023 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:01.516995 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59771a4c_e433_42eb_82c4_86e0af23ba57.slice/crio-c35c8b3fa1594441bee03426fe942175d53cc5edde664619a916c27140211bb7 WatchSource:0}: Error finding container c35c8b3fa1594441bee03426fe942175d53cc5edde664619a916c27140211bb7: Status 404 returned error can't find the container with id c35c8b3fa1594441bee03426fe942175d53cc5edde664619a916c27140211bb7 Jul 15 19:02:01.593423 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.593390 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-oauth-config\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.593423 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.593427 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-kube-api-access-pbqhx\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.593634 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.593600 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-service-ca\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.593696 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.593664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-serving-cert\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.593759 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.593716 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-config\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.593759 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.593748 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-oauth-serving-cert\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.594296 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.594272 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-service-ca\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.594404 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.594354 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-oauth-serving-cert\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.594404 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.594384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-config\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.596913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.596879 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-oauth-config\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.596913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.596895 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-serving-cert\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.604186 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.604163 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-kube-api-access-pbqhx\") pod \"console-66b74cdd84-dn5v9\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.646280 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.646233 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:01.798269 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:01.798178 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b74cdd84-dn5v9"] Jul 15 19:02:01.811141 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:01.811096 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a509a1_c7c5_4b61_a37f_5766f1daddc0.slice/crio-624e8f934a75696269e63a20265cf871af1e88502e66b2c54aae4fc58d7e1275 WatchSource:0}: Error finding container 624e8f934a75696269e63a20265cf871af1e88502e66b2c54aae4fc58d7e1275: Status 404 returned error can't find the container with id 624e8f934a75696269e63a20265cf871af1e88502e66b2c54aae4fc58d7e1275 Jul 15 19:02:02.056240 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.055954 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-9j4hf"] Jul 15 19:02:02.058516 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.058473 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.065322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.065291 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 15 19:02:02.065463 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.065395 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 15 19:02:02.065655 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.065317 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6z7rk\"" Jul 15 19:02:02.065655 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.065526 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 19:02:02.065769 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.065701 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 19:02:02.065818 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.065775 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 19:02:02.073292 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.073266 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-9j4hf"] Jul 15 19:02:02.129029 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.128983 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b74cdd84-dn5v9" event={"ID":"a5a509a1-c7c5-4b61-a37f-5766f1daddc0","Type":"ContainerStarted","Data":"624e8f934a75696269e63a20265cf871af1e88502e66b2c54aae4fc58d7e1275"} Jul 15 19:02:02.131365 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.131331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtpv" event={"ID":"59771a4c-e433-42eb-82c4-86e0af23ba57","Type":"ContainerStarted","Data":"c35c8b3fa1594441bee03426fe942175d53cc5edde664619a916c27140211bb7"} Jul 15 19:02:02.136428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.136393 2561 generic.go:356] "Generic (PLEG): container finished" podID="bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a" containerID="eb1660e5ae913c379886c61fef82a760b98cabca1a637a37912e1400e3020006" exitCode=0 Jul 15 19:02:02.136594 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.136481 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerDied","Data":"eb1660e5ae913c379886c61fef82a760b98cabca1a637a37912e1400e3020006"} Jul 15 19:02:02.140166 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.140134 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvscg" event={"ID":"0bce3637-bc3e-46ff-8456-030ce270bc5b","Type":"ContainerStarted","Data":"db6021ab624f686d9b227f30abc36664a242554cae932f05811aa023963ed906"} Jul 15 19:02:02.144823 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.144747 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fdhd" event={"ID":"9a08c025-91a2-4c90-9f14-9bc9683083d8","Type":"ContainerStarted","Data":"07989928eb2e908e7ae4790935d11b63484af793a642b2e0faba11547c7834b3"} Jul 15 19:02:02.199218 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.199173 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.199386 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.199229 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.199386 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.199369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9fh\" (UniqueName: \"kubernetes.io/projected/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-kube-api-access-cs9fh\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.199480 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.199423 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.301639 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.300494 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9fh\" (UniqueName: \"kubernetes.io/projected/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-kube-api-access-cs9fh\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.301639 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.300577 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.301639 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.300701 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.301639 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.300739 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.303740 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.302796 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-metrics-client-ca\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.310761 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.310458 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.325289 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.325259 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.332625 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.332583 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9fh\" (UniqueName: \"kubernetes.io/projected/4fc58bcb-4be3-46a3-a34a-6288a2e53bf2-kube-api-access-cs9fh\") pod \"prometheus-operator-64785b6675-9j4hf\" (UID: \"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2\") " pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.373423 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.373083 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" Jul 15 19:02:02.550707 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:02.549272 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-9j4hf"] Jul 15 19:02:02.555693 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:02.555659 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc58bcb_4be3_46a3_a34a_6288a2e53bf2.slice/crio-e4f14b8533c0e231a42cac54055ca8325ab0ede14d7e667fd349e773b0827bc2 WatchSource:0}: Error finding container e4f14b8533c0e231a42cac54055ca8325ab0ede14d7e667fd349e773b0827bc2: Status 404 returned error can't find the container with id e4f14b8533c0e231a42cac54055ca8325ab0ede14d7e667fd349e773b0827bc2 Jul 15 19:02:03.154666 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:03.153935 2561 generic.go:356] "Generic (PLEG): container finished" podID="bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a" containerID="64cfd02717648cfd6070951336689a44cc5a4d181c4d7ef4de65e2b91185580c" exitCode=0 Jul 15 19:02:03.154666 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:03.154027 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerDied","Data":"64cfd02717648cfd6070951336689a44cc5a4d181c4d7ef4de65e2b91185580c"} Jul 15 19:02:03.156830 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:03.156750 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" event={"ID":"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2","Type":"ContainerStarted","Data":"e4f14b8533c0e231a42cac54055ca8325ab0ede14d7e667fd349e773b0827bc2"} Jul 15 19:02:04.167664 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:04.167548 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jxddn" event={"ID":"bcc1ee78-8528-41f4-ac5c-0fa034ab2b9a","Type":"ContainerStarted","Data":"a46400d096c1f08f87b766fc0517a15c12f32a13e09cc57e657e3d7e1dd13a9f"} Jul 15 19:02:04.191697 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:04.191638 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jxddn" podStartSLOduration=7.323124767 podStartE2EDuration="39.191594132s" totalImagesPullingTime="31.868469365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:04.19052247 +0000 UTC m=+40.024024024" watchObservedRunningTime="2026-07-15 19:02:04.191594132 +0000 UTC m=+40.025095686" Jul 15 19:02:10.297719 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.297682 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75bb8c6d6c-dhgpw"] Jul 15 19:02:10.321514 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.321481 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bb8c6d6c-dhgpw"] Jul 15 19:02:10.321711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.321644 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.329515 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.329484 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 19:02:10.472955 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.472916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-oauth-config\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.472955 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.472950 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-trusted-ca-bundle\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.473199 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.473005 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-service-ca\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.473199 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.473052 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-config\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.473199 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.473139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jg9n\" (UniqueName: \"kubernetes.io/projected/7815e9e9-2b77-4df2-be39-15b2378fb29b-kube-api-access-8jg9n\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.473199 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.473161 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-oauth-serving-cert\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.473318 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.473181 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-serving-cert\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.574427 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.574395 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jg9n\" (UniqueName: \"kubernetes.io/projected/7815e9e9-2b77-4df2-be39-15b2378fb29b-kube-api-access-8jg9n\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.574553 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.574433 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-oauth-serving-cert\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.574553 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.574458 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-serving-cert\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.574553 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.574494 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-oauth-config\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.574553 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.574516 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-trusted-ca-bundle\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.574553 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.574550 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-service-ca\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.574817 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.574574 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-config\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.575191 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.575167 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-oauth-serving-cert\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.575252 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.575190 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-config\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.575891 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.575866 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-service-ca\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.576070 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.576048 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-trusted-ca-bundle\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.577985 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.577941 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-serving-cert\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.577985 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.577954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-oauth-config\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.591549 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.591520 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jg9n\" (UniqueName: \"kubernetes.io/projected/7815e9e9-2b77-4df2-be39-15b2378fb29b-kube-api-access-8jg9n\") pod \"console-75bb8c6d6c-dhgpw\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.633234 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.633051 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:10.942938 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:10.942893 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bb8c6d6c-dhgpw"] Jul 15 19:02:10.946419 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:10.946395 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7815e9e9_2b77_4df2_be39_15b2378fb29b.slice/crio-da62c42917e340465cfc0eeb338d5e052077a33a84dd9782cf2de5bf48b3325e WatchSource:0}: Error finding container da62c42917e340465cfc0eeb338d5e052077a33a84dd9782cf2de5bf48b3325e: Status 404 returned error can't find the container with id da62c42917e340465cfc0eeb338d5e052077a33a84dd9782cf2de5bf48b3325e Jul 15 19:02:11.189600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.189501 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvscg" event={"ID":"0bce3637-bc3e-46ff-8456-030ce270bc5b","Type":"ContainerStarted","Data":"975f4b59f40014bde2935342a559285e6a5c16a363ff771dedf102bef752fcc2"} Jul 15 19:02:11.191253 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.191220 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fdhd" event={"ID":"9a08c025-91a2-4c90-9f14-9bc9683083d8","Type":"ContainerStarted","Data":"c79f38056896463583218fb9eaa40831fc559c38de95ab5653e6a25d5387f888"} Jul 15 19:02:11.191391 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.191282 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:11.191391 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.191306 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fdhd" event={"ID":"9a08c025-91a2-4c90-9f14-9bc9683083d8","Type":"ContainerStarted","Data":"7491aa41dc8e2953a975efc1db25ba3dd4e6fc4b20a1254d4174b2774009c952"} Jul 15 19:02:11.193032 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.192972 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" event={"ID":"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2","Type":"ContainerStarted","Data":"41271e00e0b64da8dd93b29966ab6669b68b0b68248583477da88cf3876d1144"} Jul 15 19:02:11.193032 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.193018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" event={"ID":"4fc58bcb-4be3-46a3-a34a-6288a2e53bf2","Type":"ContainerStarted","Data":"9ff147b555d3b2cca37f077e0e198463aec9378b6d6d4fa67bd18dbf5c1ce7b5"} Jul 15 19:02:11.194504 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.194483 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bb8c6d6c-dhgpw" event={"ID":"7815e9e9-2b77-4df2-be39-15b2378fb29b","Type":"ContainerStarted","Data":"6f08c496ecdfdd1b1f833bcc02072bd8ee8629b7c5f4a5b92d36d1c98954cb51"} Jul 15 19:02:11.194601 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.194521 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bb8c6d6c-dhgpw" event={"ID":"7815e9e9-2b77-4df2-be39-15b2378fb29b","Type":"ContainerStarted","Data":"da62c42917e340465cfc0eeb338d5e052077a33a84dd9782cf2de5bf48b3325e"} Jul 15 19:02:11.195945 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.195924 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b74cdd84-dn5v9" event={"ID":"a5a509a1-c7c5-4b61-a37f-5766f1daddc0","Type":"ContainerStarted","Data":"cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7"} Jul 15 19:02:11.197200 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.197181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtpv" event={"ID":"59771a4c-e433-42eb-82c4-86e0af23ba57","Type":"ContainerStarted","Data":"3c2638a51a28549cafa62d277fb64b2f6c19de56f01ab3e5355666a9f8222866"} Jul 15 19:02:11.209693 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.209642 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nvscg" podStartSLOduration=2.287043496 podStartE2EDuration="11.209627097s" totalImagesPullingTime="8.922583601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:11.208372385 +0000 UTC m=+47.041873938" watchObservedRunningTime="2026-07-15 19:02:11.209627097 +0000 UTC m=+47.043128640" Jul 15 19:02:11.226949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.226891 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8fdhd" podStartSLOduration=2.304198067 podStartE2EDuration="11.22687367s" totalImagesPullingTime="8.922675603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:11.225861003 +0000 UTC m=+47.059362555" watchObservedRunningTime="2026-07-15 19:02:11.22687367 +0000 UTC m=+47.060375223" Jul 15 19:02:11.255984 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.255929 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-9j4hf" podStartSLOduration=1.36737343 podStartE2EDuration="9.255892891s" totalImagesPullingTime="7.888519461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:11.254832931 +0000 UTC m=+47.088334484" watchObservedRunningTime="2026-07-15 19:02:11.255892891 +0000 UTC m=+47.089394433" Jul 15 19:02:11.274987 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.274920 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75bb8c6d6c-dhgpw" podStartSLOduration=1.274890069 podStartE2EDuration="1.274890069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:11.274206272 +0000 UTC m=+47.107707859" watchObservedRunningTime="2026-07-15 19:02:11.274890069 +0000 UTC m=+47.108391622" Jul 15 19:02:11.292642 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.292566 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b74cdd84-dn5v9" podStartSLOduration=1.658309869 podStartE2EDuration="10.292548745s" totalImagesPullingTime="8.634238876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:11.291719228 +0000 UTC m=+47.125220776" watchObservedRunningTime="2026-07-15 19:02:11.292548745 +0000 UTC m=+47.126050297" Jul 15 19:02:11.646903 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.646863 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:11.647363 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.646943 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:11.652814 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:11.652787 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:12.207174 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:12.207143 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:02:13.363362 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.363325 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb"] Jul 15 19:02:13.525989 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.525953 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wx6vq"] Jul 15 19:02:13.526185 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.526050 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.528678 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.528656 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 19:02:13.528840 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.528815 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 19:02:13.528908 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.528855 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vmqrb\"" Jul 15 19:02:13.586459 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.586419 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-7r4q5"] Jul 15 19:02:13.586647 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.586575 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.589551 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.589412 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 19:02:13.589551 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.589438 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 19:02:13.589551 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.589451 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5shq7\"" Jul 15 19:02:13.590057 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.590037 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 19:02:13.599942 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.599910 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdxg\" (UniqueName: \"kubernetes.io/projected/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-kube-api-access-wmdxg\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.600097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.599959 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-wtmp\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600005 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzqr\" (UniqueName: \"kubernetes.io/projected/a3276f22-686f-4e8f-b0e2-b88abce74d34-kube-api-access-wqzqr\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600075 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-accelerators-collector-config\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.600338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600201 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.600338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600232 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.600338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600279 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-textfile\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600304 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-root\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600327 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-tls\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600686 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3276f22-686f-4e8f-b0e2-b88abce74d34-metrics-client-ca\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.600686 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.600385 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-sys\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.618206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.618129 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb"] Jul 15 19:02:13.618206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.618198 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-7r4q5"] Jul 15 19:02:13.618404 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.618370 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.621155 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.621097 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 19:02:13.621155 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.621145 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 19:02:13.621350 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.621155 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vw2vg\"" Jul 15 19:02:13.621350 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.621098 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 19:02:13.701357 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdxg\" (UniqueName: \"kubernetes.io/projected/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-kube-api-access-wmdxg\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.701539 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6aeae9d-e720-471a-aac8-8d3510c8bba2-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.701539 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701475 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-wtmp\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.701539 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701507 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.701539 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701532 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzqr\" (UniqueName: \"kubernetes.io/projected/a3276f22-686f-4e8f-b0e2-b88abce74d34-kube-api-access-wqzqr\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.701763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701560 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.701763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701595 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-accelerators-collector-config\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.701763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701675 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-wtmp\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.701763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.701763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701739 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.702001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701781 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dsl\" (UniqueName: \"kubernetes.io/projected/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-api-access-z8dsl\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.702001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701830 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.702001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701858 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6aeae9d-e720-471a-aac8-8d3510c8bba2-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.702001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.702001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.701956 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-textfile\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.702635 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702287 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-textfile\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.702816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702632 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-accelerators-collector-config\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.702816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702687 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.702816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702721 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-root\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.702816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702749 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-tls\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.702816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702768 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.702816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702788 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3276f22-686f-4e8f-b0e2-b88abce74d34-metrics-client-ca\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.703139 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-sys\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.703139 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702931 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-sys\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.703139 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.702972 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3276f22-686f-4e8f-b0e2-b88abce74d34-root\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.703303 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.703225 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3276f22-686f-4e8f-b0e2-b88abce74d34-metrics-client-ca\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.704715 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.704691 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.704875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.704808 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.705503 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.705469 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.705746 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.705594 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3276f22-686f-4e8f-b0e2-b88abce74d34-node-exporter-tls\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.709232 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.709207 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdxg\" (UniqueName: \"kubernetes.io/projected/b7ddf0cb-ae1e-4743-b46d-40fddf5f5247-kube-api-access-wmdxg\") pod \"openshift-state-metrics-84bd68b896-tgqtb\" (UID: \"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.709952 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.709935 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzqr\" (UniqueName: \"kubernetes.io/projected/a3276f22-686f-4e8f-b0e2-b88abce74d34-kube-api-access-wqzqr\") pod \"node-exporter-wx6vq\" (UID: \"a3276f22-686f-4e8f-b0e2-b88abce74d34\") " pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:13.803481 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.803441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dsl\" (UniqueName: \"kubernetes.io/projected/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-api-access-z8dsl\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.803481 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.803486 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6aeae9d-e720-471a-aac8-8d3510c8bba2-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.803746 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.803513 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.803746 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.803541 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.803746 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.803575 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6aeae9d-e720-471a-aac8-8d3510c8bba2-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.803746 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.803599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.803746 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:02:13.803712 2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 15 19:02:13.803982 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:02:13.803805 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-tls podName:e6aeae9d-e720-471a-aac8-8d3510c8bba2 nodeName:}" failed. No retries permitted until 2026-07-15 19:02:14.303783959 +0000 UTC m=+50.137285507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-7r4q5" (UID: "e6aeae9d-e720-471a-aac8-8d3510c8bba2") : secret "kube-state-metrics-tls" not found Jul 15 19:02:13.803982 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.803908 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6aeae9d-e720-471a-aac8-8d3510c8bba2-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.804424 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.804403 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.804477 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.804403 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6aeae9d-e720-471a-aac8-8d3510c8bba2-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.806036 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.806016 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.812794 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.812768 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dsl\" (UniqueName: \"kubernetes.io/projected/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-api-access-z8dsl\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:13.837471 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.837420 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" Jul 15 19:02:13.897530 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:13.897450 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wx6vq" Jul 15 19:02:14.309036 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.308923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:14.311656 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.311627 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6aeae9d-e720-471a-aac8-8d3510c8bba2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-7r4q5\" (UID: \"e6aeae9d-e720-471a-aac8-8d3510c8bba2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:14.462370 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.462325 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:02:14.521709 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.521665 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:02:14.521858 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.521802 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.526292 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.526254 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 19:02:14.526472 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.526460 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 19:02:14.527295 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.527281 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 19:02:14.527419 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.527390 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 19:02:14.527521 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.527505 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 19:02:14.529063 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.529044 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" Jul 15 19:02:14.536469 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.536426 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 19:02:14.536469 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.536424 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 19:02:14.536469 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.536423 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 19:02:14.536723 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.536564 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 19:02:14.536861 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.536846 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k5s6j\"" Jul 15 19:02:14.611657 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-out\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611657 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611557 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611657 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611645 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611678 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611707 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611737 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbln8\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-kube-api-access-fbln8\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611769 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.611949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.612191 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611965 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.612191 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.611986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.612191 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.612052 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-web-config\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.612191 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.612098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-volume\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.668408 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:14.668309 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3276f22_686f_4e8f_b0e2_b88abce74d34.slice/crio-7aae7ff2fe9fb7beb53e85ff17d75192f6a746ffec8db2cdc4b2e6e5b0227544 WatchSource:0}: Error finding container 7aae7ff2fe9fb7beb53e85ff17d75192f6a746ffec8db2cdc4b2e6e5b0227544: Status 404 returned error can't find the container with id 7aae7ff2fe9fb7beb53e85ff17d75192f6a746ffec8db2cdc4b2e6e5b0227544 Jul 15 19:02:14.712987 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.712855 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.712987 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.712892 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.712987 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.712915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbln8\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-kube-api-access-fbln8\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.714337 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.712946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.714337 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.713668 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.714337 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.714327 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.715017 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.714897 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.715017 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.714968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.715183 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.715029 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.716007 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.715786 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-web-config\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.716007 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.715829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-volume\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.716007 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.715872 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-out\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.716007 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.715896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.716007 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.715937 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.716434 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.716288 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.716434 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.716395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.717103 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:02:14.716810 2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 15 19:02:14.717103 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:02:14.716867 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls podName:59fc6268-8e25-45cc-bed2-92976c5e95c4 nodeName:}" failed. No retries permitted until 2026-07-15 19:02:15.216850063 +0000 UTC m=+51.050351594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4") : secret "alertmanager-main-tls" not found Jul 15 19:02:14.719125 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.719096 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.719749 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.719273 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-volume\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.719749 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.719416 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-web-config\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.719749 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.719705 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.720389 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.720340 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.720929 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.720880 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.724681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.724636 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.724963 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.724922 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-out\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.726120 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.726064 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbln8\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-kube-api-access-fbln8\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:14.810397 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.810376 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-7r4q5"] Jul 15 19:02:14.830144 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:14.830100 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb"] Jul 15 19:02:14.854483 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:14.854443 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aeae9d_e720_471a_aac8_8d3510c8bba2.slice/crio-882e6986234be0bf94d447431898f067bd2f7ffc5569e688db6bc6642d6370cd WatchSource:0}: Error finding container 882e6986234be0bf94d447431898f067bd2f7ffc5569e688db6bc6642d6370cd: Status 404 returned error can't find the container with id 882e6986234be0bf94d447431898f067bd2f7ffc5569e688db6bc6642d6370cd Jul 15 19:02:14.855487 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:14.855205 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ddf0cb_ae1e_4743_b46d_40fddf5f5247.slice/crio-a8acc18b88b84810878a852de2630e4455481c1cb001954517c62152fb49d9a3 WatchSource:0}: Error finding container a8acc18b88b84810878a852de2630e4455481c1cb001954517c62152fb49d9a3: Status 404 returned error can't find the container with id a8acc18b88b84810878a852de2630e4455481c1cb001954517c62152fb49d9a3 Jul 15 19:02:15.216445 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.216339 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtpv" event={"ID":"59771a4c-e433-42eb-82c4-86e0af23ba57","Type":"ContainerStarted","Data":"d79d06262df013dcfed572485de35758a5f286f499f25971ac49109c43904ff0"} Jul 15 19:02:15.217772 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.217740 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" event={"ID":"e6aeae9d-e720-471a-aac8-8d3510c8bba2","Type":"ContainerStarted","Data":"882e6986234be0bf94d447431898f067bd2f7ffc5569e688db6bc6642d6370cd"} Jul 15 19:02:15.219173 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.219147 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wx6vq" event={"ID":"a3276f22-686f-4e8f-b0e2-b88abce74d34","Type":"ContainerStarted","Data":"7aae7ff2fe9fb7beb53e85ff17d75192f6a746ffec8db2cdc4b2e6e5b0227544"} Jul 15 19:02:15.219481 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.219457 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:15.221312 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.221284 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" event={"ID":"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247","Type":"ContainerStarted","Data":"a25ce702822412b7540f5bc8a21a352e4ffac5a5552288c2f9e554657092a76c"} Jul 15 19:02:15.221416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.221346 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" event={"ID":"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247","Type":"ContainerStarted","Data":"55d3fb5e408a63ee21bcfeb82242a770bcec20d08757dafd76a21f7106e21142"} Jul 15 19:02:15.221416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.221376 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" event={"ID":"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247","Type":"ContainerStarted","Data":"a8acc18b88b84810878a852de2630e4455481c1cb001954517c62152fb49d9a3"} Jul 15 19:02:15.222535 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.222501 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:15.431464 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.431423 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:02:15.500790 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.500675 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d5d578769-c9q9r"] Jul 15 19:02:15.525222 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.525185 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d5d578769-c9q9r"] Jul 15 19:02:15.525448 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.525356 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.528545 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.528519 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-calvfhn7ghfk3\"" Jul 15 19:02:15.530272 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.529493 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 19:02:15.530272 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.529698 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 19:02:15.530272 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.530047 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 19:02:15.530512 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.530317 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 19:02:15.530512 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.530503 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 19:02:15.530732 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.530714 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6798b\"" Jul 15 19:02:15.624147 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624106 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587a96aa-0182-415a-b9b7-26600e6bd76d-metrics-client-ca\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.624323 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624212 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-grpc-tls\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.624323 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624285 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.624418 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624320 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.624418 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624392 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.624503 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624437 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.624503 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624466 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktzp\" (UniqueName: \"kubernetes.io/projected/587a96aa-0182-415a-b9b7-26600e6bd76d-kube-api-access-7ktzp\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.624587 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.624524 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.725470 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.725426 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.725655 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.725482 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.726042 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.725980 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktzp\" (UniqueName: \"kubernetes.io/projected/587a96aa-0182-415a-b9b7-26600e6bd76d-kube-api-access-7ktzp\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.726042 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.726032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.726149 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.726087 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587a96aa-0182-415a-b9b7-26600e6bd76d-metrics-client-ca\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.726149 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.726147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-grpc-tls\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.726246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.726168 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.726246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.726186 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.728936 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.728739 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/587a96aa-0182-415a-b9b7-26600e6bd76d-metrics-client-ca\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.734812 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.734748 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.734975 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.734908 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-grpc-tls\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.736961 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.735135 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.736961 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.736324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.736961 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.736915 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.737448 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.737422 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktzp\" (UniqueName: \"kubernetes.io/projected/587a96aa-0182-415a-b9b7-26600e6bd76d-kube-api-access-7ktzp\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.739275 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.739158 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/587a96aa-0182-415a-b9b7-26600e6bd76d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d578769-c9q9r\" (UID: \"587a96aa-0182-415a-b9b7-26600e6bd76d\") " pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.839998 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.839964 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:02:15.841431 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:15.841405 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:15.844595 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:15.844569 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fc6268_8e25_45cc_bed2_92976c5e95c4.slice/crio-af3824477d3c6b44a7fd996b5e1ba997d1d1265de4f22a04b5629311475b4b6d WatchSource:0}: Error finding container af3824477d3c6b44a7fd996b5e1ba997d1d1265de4f22a04b5629311475b4b6d: Status 404 returned error can't find the container with id af3824477d3c6b44a7fd996b5e1ba997d1d1265de4f22a04b5629311475b4b6d Jul 15 19:02:16.196968 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:16.196858 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d5d578769-c9q9r"] Jul 15 19:02:16.228115 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:16.227992 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wx6vq" event={"ID":"a3276f22-686f-4e8f-b0e2-b88abce74d34","Type":"ContainerStarted","Data":"23bada7fc4516e82b7ce5b3dd7329341e67bc4d209082c7f66c469e13a22f3e6"} Jul 15 19:02:16.230340 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:16.230303 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerStarted","Data":"af3824477d3c6b44a7fd996b5e1ba997d1d1265de4f22a04b5629311475b4b6d"} Jul 15 19:02:17.002273 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:17.002236 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587a96aa_0182_415a_b9b7_26600e6bd76d.slice/crio-e51c6f441ed8f6cc46f0f30c78ffe26ad13c468d5fbe68652b2631c05a9561ba WatchSource:0}: Error finding container e51c6f441ed8f6cc46f0f30c78ffe26ad13c468d5fbe68652b2631c05a9561ba: Status 404 returned error can't find the container with id e51c6f441ed8f6cc46f0f30c78ffe26ad13c468d5fbe68652b2631c05a9561ba Jul 15 19:02:17.234849 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.234813 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" event={"ID":"587a96aa-0182-415a-b9b7-26600e6bd76d","Type":"ContainerStarted","Data":"e51c6f441ed8f6cc46f0f30c78ffe26ad13c468d5fbe68652b2631c05a9561ba"} Jul 15 19:02:17.236233 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.236208 2561 generic.go:356] "Generic (PLEG): container finished" podID="a3276f22-686f-4e8f-b0e2-b88abce74d34" containerID="23bada7fc4516e82b7ce5b3dd7329341e67bc4d209082c7f66c469e13a22f3e6" exitCode=0 Jul 15 19:02:17.236348 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.236253 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wx6vq" event={"ID":"a3276f22-686f-4e8f-b0e2-b88abce74d34","Type":"ContainerDied","Data":"23bada7fc4516e82b7ce5b3dd7329341e67bc4d209082c7f66c469e13a22f3e6"} Jul 15 19:02:17.788955 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.788915 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8876b4966-2zksb"] Jul 15 19:02:17.831284 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.831250 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8876b4966-2zksb"] Jul 15 19:02:17.831485 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.831358 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:17.835661 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.835631 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 19:02:17.835825 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.835693 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 19:02:17.836907 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.836637 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 19:02:17.836907 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.836638 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dfti2o9jam47k\"" Jul 15 19:02:17.836907 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.836677 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 19:02:17.836907 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.836662 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vgdm9\"" Jul 15 19:02:17.948117 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.948080 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:17.948117 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.948132 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-secret-metrics-server-tls\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:17.948389 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.948256 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-client-ca-bundle\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:17.948389 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.948298 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-secret-metrics-server-client-certs\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:17.948389 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.948332 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-audit-log\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:17.948529 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.948412 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-metrics-server-audit-profiles\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:17.948529 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:17.948437 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghvg\" (UniqueName: \"kubernetes.io/projected/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-kube-api-access-rghvg\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.049544 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.049408 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.049544 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.049476 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-secret-metrics-server-tls\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.049544 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.049522 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-client-ca-bundle\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.050133 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.049547 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-secret-metrics-server-client-certs\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.050133 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.049573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-audit-log\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.050133 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.049643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-metrics-server-audit-profiles\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.050133 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.049670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghvg\" (UniqueName: \"kubernetes.io/projected/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-kube-api-access-rghvg\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.050333 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.050266 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.050390 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.050336 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-audit-log\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.050992 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.050943 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-metrics-server-audit-profiles\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.052544 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.052518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-secret-metrics-server-tls\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.052770 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.052739 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-client-ca-bundle\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.052992 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.052966 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-secret-metrics-server-client-certs\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.058236 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.058211 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghvg\" (UniqueName: \"kubernetes.io/projected/dca41ec9-d8f4-4dee-a90a-1ab6ab50d415-kube-api-access-rghvg\") pod \"metrics-server-8876b4966-2zksb\" (UID: \"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415\") " pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.143764 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.143727 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:18.151899 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.151845 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p"] Jul 15 19:02:18.218075 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.218042 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p"] Jul 15 19:02:18.218244 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.218183 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.221476 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.221451 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 19:02:18.221630 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.221518 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pljj6\"" Jul 15 19:02:18.252216 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.252178 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q2h4p\" (UID: \"9eebf6ae-0192-4b49-99ca-eba058cd5d4d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.252216 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.252225 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnc65\" (UniqueName: \"kubernetes.io/projected/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-kube-api-access-pnc65\") pod \"monitoring-plugin-5f68df7f58-q2h4p\" (UID: \"9eebf6ae-0192-4b49-99ca-eba058cd5d4d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.353568 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.353473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q2h4p\" (UID: \"9eebf6ae-0192-4b49-99ca-eba058cd5d4d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.353568 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.353517 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnc65\" (UniqueName: \"kubernetes.io/projected/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-kube-api-access-pnc65\") pod \"monitoring-plugin-5f68df7f58-q2h4p\" (UID: \"9eebf6ae-0192-4b49-99ca-eba058cd5d4d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.353813 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:02:18.353671 2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 15 19:02:18.353813 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:02:18.353755 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-monitoring-plugin-cert podName:9eebf6ae-0192-4b49-99ca-eba058cd5d4d nodeName:}" failed. No retries permitted until 2026-07-15 19:02:18.853730666 +0000 UTC m=+54.687232214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-q2h4p" (UID: "9eebf6ae-0192-4b49-99ca-eba058cd5d4d") : secret "monitoring-plugin-cert" not found Jul 15 19:02:18.364816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.364786 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnc65\" (UniqueName: \"kubernetes.io/projected/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-kube-api-access-pnc65\") pod \"monitoring-plugin-5f68df7f58-q2h4p\" (UID: \"9eebf6ae-0192-4b49-99ca-eba058cd5d4d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.637028 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.636946 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6df798c7c-n4dtz"] Jul 15 19:02:18.646035 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.646001 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.649097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.648743 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4n9n4\"" Jul 15 19:02:18.649097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.648770 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 19:02:18.649097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.648813 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 19:02:18.649097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.648854 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 19:02:18.649097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.648877 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 19:02:18.649531 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.649206 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 19:02:18.655070 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.655000 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6df798c7c-n4dtz"] Jul 15 19:02:18.655266 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.655246 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 19:02:18.757080 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-telemeter-client-tls\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.757080 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757080 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtrxk\" (UniqueName: \"kubernetes.io/projected/885483b6-feb3-4643-9240-86e8d807fb5f-kube-api-access-qtrxk\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.757316 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757116 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-secret-telemeter-client\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.757316 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757165 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-serving-certs-ca-bundle\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.757316 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757197 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-metrics-client-ca\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.757438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757344 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.757438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757403 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-federate-client-tls\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.757515 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.757436 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858054 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858015 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-telemeter-client-tls\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858054 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858061 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtrxk\" (UniqueName: \"kubernetes.io/projected/885483b6-feb3-4643-9240-86e8d807fb5f-kube-api-access-qtrxk\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858290 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858089 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-secret-telemeter-client\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858290 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-serving-certs-ca-bundle\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858290 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858141 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-metrics-client-ca\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858290 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858215 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q2h4p\" (UID: \"9eebf6ae-0192-4b49-99ca-eba058cd5d4d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.858290 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858244 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858290 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858291 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-federate-client-tls\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.858564 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.858320 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.859194 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.859080 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-serving-certs-ca-bundle\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.859940 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.859911 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-metrics-client-ca\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.860247 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.860215 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/885483b6-feb3-4643-9240-86e8d807fb5f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.861174 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.861143 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.861619 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.861581 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eebf6ae-0192-4b49-99ca-eba058cd5d4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q2h4p\" (UID: \"9eebf6ae-0192-4b49-99ca-eba058cd5d4d\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:18.861619 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.861594 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-telemeter-client-tls\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.862038 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.862020 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-secret-telemeter-client\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.862502 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.862475 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/885483b6-feb3-4643-9240-86e8d807fb5f-federate-client-tls\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.866912 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.866889 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtrxk\" (UniqueName: \"kubernetes.io/projected/885483b6-feb3-4643-9240-86e8d807fb5f-kube-api-access-qtrxk\") pod \"telemeter-client-6df798c7c-n4dtz\" (UID: \"885483b6-feb3-4643-9240-86e8d807fb5f\") " pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:18.958989 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:18.958900 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" Jul 15 19:02:19.129348 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.129320 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:19.203904 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.203747 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8876b4966-2zksb"] Jul 15 19:02:19.227329 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.227239 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6df798c7c-n4dtz"] Jul 15 19:02:19.370195 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:19.370160 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca41ec9_d8f4_4dee_a90a_1ab6ab50d415.slice/crio-5e9f6643e2793deb37fe02984d3aa95249e0e39038f45241b10edc8c8494f969 WatchSource:0}: Error finding container 5e9f6643e2793deb37fe02984d3aa95249e0e39038f45241b10edc8c8494f969: Status 404 returned error can't find the container with id 5e9f6643e2793deb37fe02984d3aa95249e0e39038f45241b10edc8c8494f969 Jul 15 19:02:19.644539 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.644450 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p"] Jul 15 19:02:19.827455 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.827275 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:02:19.833169 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.833131 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.835743 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.835707 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rhftj\"" Jul 15 19:02:19.837891 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.836958 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fso41rpjh0g74\"" Jul 15 19:02:19.837891 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.837237 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 19:02:19.837891 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.837457 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 19:02:19.837891 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.837741 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 19:02:19.848163 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.848129 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 19:02:19.848163 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.848153 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 19:02:19.848362 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.848202 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 19:02:19.848671 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.848653 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 19:02:19.849390 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.849370 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 19:02:19.850550 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.850341 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 19:02:19.850550 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.850379 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 19:02:19.850550 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.850417 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 19:02:19.851053 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.851036 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 19:02:19.852534 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.852514 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 19:02:19.858767 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.858736 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 19:02:19.859742 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.859714 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:02:19.874247 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874213 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874247 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874248 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874325 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-config-out\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t77x\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-kube-api-access-8t77x\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874492 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-web-config\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874533 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874579 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874663 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874690 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874734 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.874785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874782 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.875169 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-config\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.875169 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874879 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.875169 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874932 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.875169 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.874960 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.875169 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.875024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.975970 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.975935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-config\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976140 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.975987 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976140 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976140 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976140 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976358 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976143 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976358 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976168 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976358 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976204 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976358 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976236 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-config-out\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976358 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976268 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t77x\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-kube-api-access-8t77x\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976358 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976302 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976358 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976351 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976407 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-web-config\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976577 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976621 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976649 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.976726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.976672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.977331 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.977306 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.978327 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.978196 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.984118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.980820 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.984118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.981133 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.984118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.982063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.984118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.982953 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-config\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.990668 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.990640 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.994816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.993023 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.994816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.994250 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.994816 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.994279 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-config-out\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.996533 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.996496 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.996651 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.996598 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.997196 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.997172 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.998374 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.998331 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-web-config\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.998518 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.998494 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.998927 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.998896 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.999439 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.999081 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:19.999847 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:19.999797 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:20.001806 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.001769 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:20.002271 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.002228 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t77x\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-kube-api-access-8t77x\") pod \"prometheus-k8s-0\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:20.145065 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.144964 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:20.265456 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.264725 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" event={"ID":"e6aeae9d-e720-471a-aac8-8d3510c8bba2","Type":"ContainerStarted","Data":"5ef7a62df9816c4e18113e33586cf8d7c14aace35b9d425ab30b623594173538"} Jul 15 19:02:20.265456 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.264839 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" event={"ID":"e6aeae9d-e720-471a-aac8-8d3510c8bba2","Type":"ContainerStarted","Data":"19cbcf504849ab129cd7b8700b1d824682c0551b38eca7771d31a706f7ba4927"} Jul 15 19:02:20.265456 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.264871 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" event={"ID":"e6aeae9d-e720-471a-aac8-8d3510c8bba2","Type":"ContainerStarted","Data":"068886a33180da7779903505d34d047dd8408fd528659b871e39c01457f321a4"} Jul 15 19:02:20.271673 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.270946 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wx6vq" event={"ID":"a3276f22-686f-4e8f-b0e2-b88abce74d34","Type":"ContainerStarted","Data":"07d5a806254850ceaa51f160d2bc7f996491026bacfbe74285f95f3afd54b647"} Jul 15 19:02:20.271673 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.271010 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wx6vq" event={"ID":"a3276f22-686f-4e8f-b0e2-b88abce74d34","Type":"ContainerStarted","Data":"d4640e888b82acb2c3fb3bcea72f73a876ecdfed3355c9171c2d567bf8b99c95"} Jul 15 19:02:20.274988 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.274008 2561 generic.go:356] "Generic (PLEG): container finished" podID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerID="2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a" exitCode=0 Jul 15 19:02:20.274988 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.274360 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a"} Jul 15 19:02:20.280100 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.280019 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" event={"ID":"b7ddf0cb-ae1e-4743-b46d-40fddf5f5247","Type":"ContainerStarted","Data":"43aa6626a9303a95b06c662cd2c93a19d397932225a43e6e5b1910fcd1fa2920"} Jul 15 19:02:20.283418 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.283388 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" event={"ID":"9eebf6ae-0192-4b49-99ca-eba058cd5d4d","Type":"ContainerStarted","Data":"3a34d7ae05c98d5e08ca5074d1c6fc485a3f34b12aee631f8f72e256184f774a"} Jul 15 19:02:20.291670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.291634 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" event={"ID":"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415","Type":"ContainerStarted","Data":"5e9f6643e2793deb37fe02984d3aa95249e0e39038f45241b10edc8c8494f969"} Jul 15 19:02:20.294831 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.294798 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtpv" event={"ID":"59771a4c-e433-42eb-82c4-86e0af23ba57","Type":"ContainerStarted","Data":"31a7ae9b7f979e1de3999b71ea1e7d1d40662b217aab3bb0bfb80c44dcff240f"} Jul 15 19:02:20.298831 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.298778 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" event={"ID":"885483b6-feb3-4643-9240-86e8d807fb5f","Type":"ContainerStarted","Data":"5bfeec787e66b28bc4c247b1414c237fd15dc5332d6fd1bb67e34b786eb85a71"} Jul 15 19:02:20.320124 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.318512 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wx6vq" podStartSLOduration=5.811416003 podStartE2EDuration="7.318493791s" totalImagesPullingTime="1.373314879s" totalInitContainerRuntime="133.762909ms" isStatefulPod=true podCreationTimestamp="2026-07-15 19:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:20.31659858 +0000 UTC m=+56.150100154" watchObservedRunningTime="2026-07-15 19:02:20.318493791 +0000 UTC m=+56.151995347" Jul 15 19:02:20.320124 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.319573 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-7r4q5" podStartSLOduration=3.136147362 podStartE2EDuration="7.319557605s" totalImagesPullingTime="4.183410243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:20.291956321 +0000 UTC m=+56.125457874" watchObservedRunningTime="2026-07-15 19:02:20.319557605 +0000 UTC m=+56.153059159" Jul 15 19:02:20.336814 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.336782 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:02:20.341161 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:20.341134 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c0c8f8_86bf_4172_855b_9c57495d931e.slice/crio-a2c9849aaf6450e2125fb44c7ba113b1a7d0729a47c0568c1c9034fa92f9c731 WatchSource:0}: Error finding container a2c9849aaf6450e2125fb44c7ba113b1a7d0729a47c0568c1c9034fa92f9c731: Status 404 returned error can't find the container with id a2c9849aaf6450e2125fb44c7ba113b1a7d0729a47c0568c1c9034fa92f9c731 Jul 15 19:02:20.388518 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.388450 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tgqtb" podStartSLOduration=3.3857227549999998 podStartE2EDuration="7.388427795s" totalImagesPullingTime="4.00270504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:20.386599861 +0000 UTC m=+56.220101415" watchObservedRunningTime="2026-07-15 19:02:20.388427795 +0000 UTC m=+56.221929340" Jul 15 19:02:20.412852 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.412621 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7vtpv" podStartSLOduration=2.757946218 podStartE2EDuration="20.412577103s" totalImagesPullingTime="17.654630885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:02:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:20.412378941 +0000 UTC m=+56.245880508" watchObservedRunningTime="2026-07-15 19:02:20.412577103 +0000 UTC m=+56.246078655" Jul 15 19:02:20.633800 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.633761 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:20.633977 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.633844 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:20.640397 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.640341 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:20.729275 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:20.729147 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75bb8c6d6c-dhgpw"] Jul 15 19:02:21.204190 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:21.204152 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8fdhd" Jul 15 19:02:21.306769 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:21.306733 2561 generic.go:356] "Generic (PLEG): container finished" podID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerID="cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079" exitCode=0 Jul 15 19:02:21.306949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:21.306818 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079"} Jul 15 19:02:21.306949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:21.306917 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerStarted","Data":"a2c9849aaf6450e2125fb44c7ba113b1a7d0729a47c0568c1c9034fa92f9c731"} Jul 15 19:02:21.313701 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:21.313671 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:24.323169 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.323058 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" event={"ID":"587a96aa-0182-415a-b9b7-26600e6bd76d","Type":"ContainerStarted","Data":"c6fffcfcbb3fccd9fdd22c6ef8c6b5ed0a18ebb517768810c758d4588bd714d6"} Jul 15 19:02:24.323627 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.323268 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" event={"ID":"587a96aa-0182-415a-b9b7-26600e6bd76d","Type":"ContainerStarted","Data":"5b83991bac020e74ef91235e321343bc451958fc1e8d0a04f2440d8a0f72de3a"} Jul 15 19:02:24.323627 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.323316 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" event={"ID":"587a96aa-0182-415a-b9b7-26600e6bd76d","Type":"ContainerStarted","Data":"23eca7e827a970f62cdd8abc8893bac20445be44f1ad3821e5b2f292b02c86df"} Jul 15 19:02:24.324843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.324810 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" event={"ID":"dca41ec9-d8f4-4dee-a90a-1ab6ab50d415","Type":"ContainerStarted","Data":"891f6634c25aa3fc108182c02a0349a0cdcdf5645de70e3467640518c4d3ae1d"} Jul 15 19:02:24.327466 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.327443 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" event={"ID":"885483b6-feb3-4643-9240-86e8d807fb5f","Type":"ContainerStarted","Data":"ca718c904f7a68953a0a717779c9b7754dfe58bf6a6d61c23999823f5a8d61fe"} Jul 15 19:02:24.327595 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.327496 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" event={"ID":"885483b6-feb3-4643-9240-86e8d807fb5f","Type":"ContainerStarted","Data":"71f0055e9a2c115895b1ed62425bb25984ba15c8887878c0794fa7f5f279f56b"} Jul 15 19:02:24.327595 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.327528 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" event={"ID":"885483b6-feb3-4643-9240-86e8d807fb5f","Type":"ContainerStarted","Data":"3c3a76ab9e3b85aa00ad6a26e4ee2f00be6525f2135a14dd83373a674845b2f6"} Jul 15 19:02:24.330649 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.330600 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerStarted","Data":"3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878"} Jul 15 19:02:24.330785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.330692 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerStarted","Data":"345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63"} Jul 15 19:02:24.330785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.330732 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerStarted","Data":"ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0"} Jul 15 19:02:24.332224 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.332202 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" event={"ID":"9eebf6ae-0192-4b49-99ca-eba058cd5d4d","Type":"ContainerStarted","Data":"9e1d16b36954eb638f216ae888fa26f552301032d9c1e848ebb27f9c142d7ba8"} Jul 15 19:02:24.332432 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.332415 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:24.338686 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.338665 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" Jul 15 19:02:24.345446 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.345339 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" podStartSLOduration=3.027168275 podStartE2EDuration="7.34532108s" totalImagesPullingTime="4.318152805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:24.34519658 +0000 UTC m=+60.178698133" watchObservedRunningTime="2026-07-15 19:02:24.34532108 +0000 UTC m=+60.178822633" Jul 15 19:02:24.371158 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.371035 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6df798c7c-n4dtz" podStartSLOduration=2.04746458 podStartE2EDuration="6.371018485s" totalImagesPullingTime="4.323553905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:24.368821613 +0000 UTC m=+60.202323179" watchObservedRunningTime="2026-07-15 19:02:24.371018485 +0000 UTC m=+60.204520037" Jul 15 19:02:24.393148 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:24.393087 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q2h4p" podStartSLOduration=2.374419018 podStartE2EDuration="6.393064758s" totalImagesPullingTime="4.01864574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:24.39229806 +0000 UTC m=+60.225799637" watchObservedRunningTime="2026-07-15 19:02:24.393064758 +0000 UTC m=+60.226566306" Jul 15 19:02:25.340275 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:25.340232 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerStarted","Data":"a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32"} Jul 15 19:02:25.340754 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:25.340322 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerStarted","Data":"903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7"} Jul 15 19:02:26.112101 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:26.112074 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kc4vn" Jul 15 19:02:26.353075 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:26.352883 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" event={"ID":"587a96aa-0182-415a-b9b7-26600e6bd76d","Type":"ContainerStarted","Data":"56e567e398949f80f3fb3de3c3bc178bb78ec004f4e1d31fc2b65e16a36bf9cd"} Jul 15 19:02:26.353075 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:26.353006 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" event={"ID":"587a96aa-0182-415a-b9b7-26600e6bd76d","Type":"ContainerStarted","Data":"f513f821dc1d0e82aecdc9a945eb580d67a0a472f1b1a18755049a114378fb38"} Jul 15 19:02:26.358648 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:26.358599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerStarted","Data":"96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038"} Jul 15 19:02:26.363385 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:26.362828 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerStarted","Data":"1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27"} Jul 15 19:02:26.363385 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:26.362961 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerStarted","Data":"1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc"} Jul 15 19:02:27.370759 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.370721 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerStarted","Data":"980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c"} Jul 15 19:02:27.371198 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.370796 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerStarted","Data":"82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca"} Jul 15 19:02:27.371198 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.370825 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerStarted","Data":"d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5"} Jul 15 19:02:27.371198 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.370858 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerStarted","Data":"b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557"} Jul 15 19:02:27.373779 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.373748 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" event={"ID":"587a96aa-0182-415a-b9b7-26600e6bd76d","Type":"ContainerStarted","Data":"be7378fcf7df6895b9b00d085eaab4fe74411ab2d77370faecaaa58cfcfd4f21"} Jul 15 19:02:27.398703 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.398651 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.556172687 podStartE2EDuration="8.398627836s" totalImagesPullingTime="4.810090703s" totalInitContainerRuntime="32.364446ms" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:27.396998835 +0000 UTC m=+63.230500416" watchObservedRunningTime="2026-07-15 19:02:27.398627836 +0000 UTC m=+63.232129384" Jul 15 19:02:27.399382 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.399338 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.737557439 podStartE2EDuration="13.399324058s" totalImagesPullingTime="8.632385632s" totalInitContainerRuntime="29.380987ms" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:26.395038686 +0000 UTC m=+62.228540296" watchObservedRunningTime="2026-07-15 19:02:27.399324058 +0000 UTC m=+63.232825613" Jul 15 19:02:27.418756 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:27.418708 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" podStartSLOduration=4.073024195 podStartE2EDuration="12.418688493s" totalImagesPullingTime="8.345664298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:02:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:27.416538548 +0000 UTC m=+63.250040140" watchObservedRunningTime="2026-07-15 19:02:27.418688493 +0000 UTC m=+63.252190045" Jul 15 19:02:28.378868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:28.378830 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:28.384118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:28.384095 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d5d578769-c9q9r" Jul 15 19:02:30.145310 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.145269 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:30.495458 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.495354 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:02:30.495731 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.495702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:02:30.498009 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.497992 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:02:30.498866 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.498852 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:02:30.508834 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.508804 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/398e6ee4-ffb7-418d-a5f1-31b4528fcae7-metrics-certs\") pod \"network-metrics-daemon-skp8g\" (UID: \"398e6ee4-ffb7-418d-a5f1-31b4528fcae7\") " pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:02:30.508930 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.508873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/707f404d-e473-4ea4-99e1-3bfb155dfd23-original-pull-secret\") pod \"global-pull-secret-syncer-fxjc5\" (UID: \"707f404d-e473-4ea4-99e1-3bfb155dfd23\") " pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:02:30.596533 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.596493 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:02:30.599140 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.599121 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:02:30.609488 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.609466 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:02:30.619973 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.619952 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppq4\" (UniqueName: \"kubernetes.io/projected/cd2b941e-8f72-40a0-bdd7-c50333aebbc4-kube-api-access-4ppq4\") pod \"network-check-target-4hlkp\" (UID: \"cd2b941e-8f72-40a0-bdd7-c50333aebbc4\") " pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:02:30.642725 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.642699 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7hbwd\"" Jul 15 19:02:30.649182 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.649159 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j7rjq\"" Jul 15 19:02:30.650161 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.650141 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-skp8g" Jul 15 19:02:30.653336 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.653315 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxjc5" Jul 15 19:02:30.657688 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.657669 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:02:30.810063 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:30.810031 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxjc5"] Jul 15 19:02:30.815393 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:30.815363 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707f404d_e473_4ea4_99e1_3bfb155dfd23.slice/crio-e2c9058ad3bbbc68734253a39f7a80e2925ad69ba093a3bde7081bc65e02973c WatchSource:0}: Error finding container e2c9058ad3bbbc68734253a39f7a80e2925ad69ba093a3bde7081bc65e02973c: Status 404 returned error can't find the container with id e2c9058ad3bbbc68734253a39f7a80e2925ad69ba093a3bde7081bc65e02973c Jul 15 19:02:31.042859 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:31.042763 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4hlkp"] Jul 15 19:02:31.049326 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:31.049262 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-skp8g"] Jul 15 19:02:31.053806 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:02:31.053784 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398e6ee4_ffb7_418d_a5f1_31b4528fcae7.slice/crio-adf2061c696c4a790f1d3789efe704c9bc339f99d6d28fabe1cf75f13149c991 WatchSource:0}: Error finding container adf2061c696c4a790f1d3789efe704c9bc339f99d6d28fabe1cf75f13149c991: Status 404 returned error can't find the container with id adf2061c696c4a790f1d3789efe704c9bc339f99d6d28fabe1cf75f13149c991 Jul 15 19:02:31.391952 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:31.391872 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4hlkp" event={"ID":"cd2b941e-8f72-40a0-bdd7-c50333aebbc4","Type":"ContainerStarted","Data":"c56ba3bc32d70f609a1d7c0d70be839c9205863d5be4d15934000592973aa22e"} Jul 15 19:02:31.392961 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:31.392938 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxjc5" event={"ID":"707f404d-e473-4ea4-99e1-3bfb155dfd23","Type":"ContainerStarted","Data":"e2c9058ad3bbbc68734253a39f7a80e2925ad69ba093a3bde7081bc65e02973c"} Jul 15 19:02:31.393995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:31.393974 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skp8g" event={"ID":"398e6ee4-ffb7-418d-a5f1-31b4528fcae7","Type":"ContainerStarted","Data":"adf2061c696c4a790f1d3789efe704c9bc339f99d6d28fabe1cf75f13149c991"} Jul 15 19:02:32.401436 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:32.401335 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skp8g" event={"ID":"398e6ee4-ffb7-418d-a5f1-31b4528fcae7","Type":"ContainerStarted","Data":"621c6917a2786a66f7943f20cfd149869c469e7b95b4602f94ae1a0e153fb50f"} Jul 15 19:02:32.401436 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:32.401408 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-skp8g" event={"ID":"398e6ee4-ffb7-418d-a5f1-31b4528fcae7","Type":"ContainerStarted","Data":"8fa6129d303c4ae4958c052c1dbd560e995a7e7ddb7b552b656640c3f86cce5d"} Jul 15 19:02:32.420465 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:32.420413 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-skp8g" podStartSLOduration=66.408177663 podStartE2EDuration="1m7.420390413s" totalImagesPullingTime="1.01221275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:32.419295879 +0000 UTC m=+68.252797432" watchObservedRunningTime="2026-07-15 19:02:32.420390413 +0000 UTC m=+68.253891966" Jul 15 19:02:34.416406 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:34.416369 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4hlkp" event={"ID":"cd2b941e-8f72-40a0-bdd7-c50333aebbc4","Type":"ContainerStarted","Data":"3c2f65c492196a7a0dbdfa0c52a8e350ee3ed897df0bab4ac0f300ba31aa038c"} Jul 15 19:02:34.416895 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:34.416480 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:02:34.438713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:34.438644 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4hlkp" podStartSLOduration=66.516254129 podStartE2EDuration="1m9.438602614s" totalImagesPullingTime="2.922348485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:34.437021827 +0000 UTC m=+70.270523379" watchObservedRunningTime="2026-07-15 19:02:34.438602614 +0000 UTC m=+70.272104167" Jul 15 19:02:35.766126 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:35.766095 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b74cdd84-dn5v9"] Jul 15 19:02:38.144569 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:38.144534 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:38.144988 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:38.144598 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:38.437020 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:38.436935 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxjc5" event={"ID":"707f404d-e473-4ea4-99e1-3bfb155dfd23","Type":"ContainerStarted","Data":"adc36f87d65d7e2efe1eaa87e7d070034c32f3277d30abe9f5534f946e7441d6"} Jul 15 19:02:38.454347 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:38.454296 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fxjc5" podStartSLOduration=66.022602363 podStartE2EDuration="1m13.454278292s" totalImagesPullingTime="7.431675929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:02:38.452985039 +0000 UTC m=+74.286486604" watchObservedRunningTime="2026-07-15 19:02:38.454278292 +0000 UTC m=+74.287779843" Jul 15 19:02:47.333086 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.332920 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75bb8c6d6c-dhgpw" podUID="7815e9e9-2b77-4df2-be39-15b2378fb29b" containerName="console" containerID="cri-o://6f08c496ecdfdd1b1f833bcc02072bd8ee8629b7c5f4a5b92d36d1c98954cb51" gracePeriod=15 Jul 15 19:02:47.481188 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.481158 2561 generic.go:356] "Generic (PLEG): container finished" podID="7815e9e9-2b77-4df2-be39-15b2378fb29b" containerID="6f08c496ecdfdd1b1f833bcc02072bd8ee8629b7c5f4a5b92d36d1c98954cb51" exitCode=0 Jul 15 19:02:47.481302 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.481237 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bb8c6d6c-dhgpw" event={"ID":"7815e9e9-2b77-4df2-be39-15b2378fb29b","Type":"ContainerDied","Data":"6f08c496ecdfdd1b1f833bcc02072bd8ee8629b7c5f4a5b92d36d1c98954cb51"} Jul 15 19:02:47.640414 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.640381 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:47.662922 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.662893 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-trusted-ca-bundle\") pod \"7815e9e9-2b77-4df2-be39-15b2378fb29b\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " Jul 15 19:02:47.663106 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.662958 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-oauth-config\") pod \"7815e9e9-2b77-4df2-be39-15b2378fb29b\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " Jul 15 19:02:47.663106 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.662999 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jg9n\" (UniqueName: \"kubernetes.io/projected/7815e9e9-2b77-4df2-be39-15b2378fb29b-kube-api-access-8jg9n\") pod \"7815e9e9-2b77-4df2-be39-15b2378fb29b\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " Jul 15 19:02:47.663228 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.663143 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-service-ca\") pod \"7815e9e9-2b77-4df2-be39-15b2378fb29b\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " Jul 15 19:02:47.663228 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.663203 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-config\") pod \"7815e9e9-2b77-4df2-be39-15b2378fb29b\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " Jul 15 19:02:47.663343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.663228 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-oauth-serving-cert\") pod \"7815e9e9-2b77-4df2-be39-15b2378fb29b\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " Jul 15 19:02:47.663343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.663253 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-serving-cert\") pod \"7815e9e9-2b77-4df2-be39-15b2378fb29b\" (UID: \"7815e9e9-2b77-4df2-be39-15b2378fb29b\") " Jul 15 19:02:47.663343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.663300 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7815e9e9-2b77-4df2-be39-15b2378fb29b" (UID: "7815e9e9-2b77-4df2-be39-15b2378fb29b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:02:47.663564 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.663543 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-trusted-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:02:47.664151 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.664093 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-service-ca" (OuterVolumeSpecName: "service-ca") pod "7815e9e9-2b77-4df2-be39-15b2378fb29b" (UID: "7815e9e9-2b77-4df2-be39-15b2378fb29b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:02:47.664259 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.664178 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-config" (OuterVolumeSpecName: "console-config") pod "7815e9e9-2b77-4df2-be39-15b2378fb29b" (UID: "7815e9e9-2b77-4df2-be39-15b2378fb29b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:02:47.664830 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.664533 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7815e9e9-2b77-4df2-be39-15b2378fb29b" (UID: "7815e9e9-2b77-4df2-be39-15b2378fb29b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:02:47.666391 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.666361 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7815e9e9-2b77-4df2-be39-15b2378fb29b" (UID: "7815e9e9-2b77-4df2-be39-15b2378fb29b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:02:47.666487 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.666448 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7815e9e9-2b77-4df2-be39-15b2378fb29b" (UID: "7815e9e9-2b77-4df2-be39-15b2378fb29b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:02:47.666799 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.666770 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7815e9e9-2b77-4df2-be39-15b2378fb29b-kube-api-access-8jg9n" (OuterVolumeSpecName: "kube-api-access-8jg9n") pod "7815e9e9-2b77-4df2-be39-15b2378fb29b" (UID: "7815e9e9-2b77-4df2-be39-15b2378fb29b"). InnerVolumeSpecName "kube-api-access-8jg9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:02:47.764031 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.763986 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:02:47.764031 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.764018 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-oauth-serving-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:02:47.764031 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.764027 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-serving-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:02:47.764335 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.764045 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7815e9e9-2b77-4df2-be39-15b2378fb29b-console-oauth-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:02:47.764335 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.764056 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jg9n\" (UniqueName: \"kubernetes.io/projected/7815e9e9-2b77-4df2-be39-15b2378fb29b-kube-api-access-8jg9n\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:02:47.764335 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:47.764066 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7815e9e9-2b77-4df2-be39-15b2378fb29b-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:02:48.486883 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:48.486858 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bb8c6d6c-dhgpw" Jul 15 19:02:48.486883 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:48.486867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bb8c6d6c-dhgpw" event={"ID":"7815e9e9-2b77-4df2-be39-15b2378fb29b","Type":"ContainerDied","Data":"da62c42917e340465cfc0eeb338d5e052077a33a84dd9782cf2de5bf48b3325e"} Jul 15 19:02:48.495463 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:48.495444 2561 scope.go:117] "RemoveContainer" containerID="6f08c496ecdfdd1b1f833bcc02072bd8ee8629b7c5f4a5b92d36d1c98954cb51" Jul 15 19:02:48.507416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:48.507392 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75bb8c6d6c-dhgpw"] Jul 15 19:02:48.511454 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:48.511432 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75bb8c6d6c-dhgpw"] Jul 15 19:02:48.931495 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:48.931459 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7815e9e9-2b77-4df2-be39-15b2378fb29b" path="/var/lib/kubelet/pods/7815e9e9-2b77-4df2-be39-15b2378fb29b/volumes" Jul 15 19:02:55.203437 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:55.203406 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:55.263165 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:55.263129 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:55.531560 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:55.531534 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:02:58.154143 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:58.154108 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:02:58.161861 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:02:58.161829 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8876b4966-2zksb" Jul 15 19:03:00.786908 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:00.786845 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66b74cdd84-dn5v9" podUID="a5a509a1-c7c5-4b61-a37f-5766f1daddc0" containerName="console" containerID="cri-o://cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7" gracePeriod=15 Jul 15 19:03:01.035036 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.035010 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:03:01.090322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090234 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-oauth-config\") pod \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " Jul 15 19:03:01.090322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090276 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-kube-api-access-pbqhx\") pod \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " Jul 15 19:03:01.090322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090306 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-oauth-serving-cert\") pod \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " Jul 15 19:03:01.090878 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090436 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-serving-cert\") pod \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " Jul 15 19:03:01.090878 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090494 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-config\") pod \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " Jul 15 19:03:01.090878 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090524 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-service-ca\") pod \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\" (UID: \"a5a509a1-c7c5-4b61-a37f-5766f1daddc0\") " Jul 15 19:03:01.090878 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090832 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a5a509a1-c7c5-4b61-a37f-5766f1daddc0" (UID: "a5a509a1-c7c5-4b61-a37f-5766f1daddc0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:01.091063 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090989 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-service-ca" (OuterVolumeSpecName: "service-ca") pod "a5a509a1-c7c5-4b61-a37f-5766f1daddc0" (UID: "a5a509a1-c7c5-4b61-a37f-5766f1daddc0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:01.091063 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.090996 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-config" (OuterVolumeSpecName: "console-config") pod "a5a509a1-c7c5-4b61-a37f-5766f1daddc0" (UID: "a5a509a1-c7c5-4b61-a37f-5766f1daddc0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:01.092655 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.092633 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a5a509a1-c7c5-4b61-a37f-5766f1daddc0" (UID: "a5a509a1-c7c5-4b61-a37f-5766f1daddc0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:01.093061 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.093028 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-kube-api-access-pbqhx" (OuterVolumeSpecName: "kube-api-access-pbqhx") pod "a5a509a1-c7c5-4b61-a37f-5766f1daddc0" (UID: "a5a509a1-c7c5-4b61-a37f-5766f1daddc0"). InnerVolumeSpecName "kube-api-access-pbqhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:03:01.093061 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.093028 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a5a509a1-c7c5-4b61-a37f-5766f1daddc0" (UID: "a5a509a1-c7c5-4b61-a37f-5766f1daddc0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:01.191886 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.191847 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-serving-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:01.191886 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.191878 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:01.191886 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.191888 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-service-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:01.191886 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.191897 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-console-oauth-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:01.192151 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.191907 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqhx\" (UniqueName: \"kubernetes.io/projected/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-kube-api-access-pbqhx\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:01.192151 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.191916 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5a509a1-c7c5-4b61-a37f-5766f1daddc0-oauth-serving-cert\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:01.544438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.544406 2561 generic.go:356] "Generic (PLEG): container finished" podID="a5a509a1-c7c5-4b61-a37f-5766f1daddc0" containerID="cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7" exitCode=0 Jul 15 19:03:01.544600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.544457 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b74cdd84-dn5v9" Jul 15 19:03:01.544600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.544491 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b74cdd84-dn5v9" event={"ID":"a5a509a1-c7c5-4b61-a37f-5766f1daddc0","Type":"ContainerDied","Data":"cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7"} Jul 15 19:03:01.544600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.544549 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b74cdd84-dn5v9" event={"ID":"a5a509a1-c7c5-4b61-a37f-5766f1daddc0","Type":"ContainerDied","Data":"624e8f934a75696269e63a20265cf871af1e88502e66b2c54aae4fc58d7e1275"} Jul 15 19:03:01.544600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.544575 2561 scope.go:117] "RemoveContainer" containerID="cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7" Jul 15 19:03:01.553674 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.553654 2561 scope.go:117] "RemoveContainer" containerID="cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7" Jul 15 19:03:01.553944 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:01.553927 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7\": container with ID starting with cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7 not found: ID does not exist" containerID="cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7" Jul 15 19:03:01.553988 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.553953 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7"} err="failed to get container status \"cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7\": rpc error: code = NotFound desc = could not find container \"cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7\": container with ID starting with cc9b7b7655979469c4b2bda1d9cec39b23a2881812d6388a4fd71b87af5611b7 not found: ID does not exist" Jul 15 19:03:01.572660 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.570293 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b74cdd84-dn5v9"] Jul 15 19:03:01.575008 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:01.574985 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66b74cdd84-dn5v9"] Jul 15 19:03:02.934197 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:02.934160 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a509a1-c7c5-4b61-a37f-5766f1daddc0" path="/var/lib/kubelet/pods/a5a509a1-c7c5-4b61-a37f-5766f1daddc0/volumes" Jul 15 19:03:03.766207 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:03.766173 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:03:03.766500 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:03.766478 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="alertmanager" containerID="cri-o://ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0" gracePeriod=120 Jul 15 19:03:03.766583 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:03.766556 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-metric" containerID="cri-o://a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32" gracePeriod=120 Jul 15 19:03:03.766672 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:03.766584 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="config-reloader" containerID="cri-o://345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63" gracePeriod=120 Jul 15 19:03:03.766724 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:03.766639 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy" containerID="cri-o://903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7" gracePeriod=120 Jul 15 19:03:03.766724 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:03.766568 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-web" containerID="cri-o://3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878" gracePeriod=120 Jul 15 19:03:03.766724 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:03.766594 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="prom-label-proxy" containerID="cri-o://96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038" gracePeriod=120 Jul 15 19:03:04.561197 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561166 2561 generic.go:356] "Generic (PLEG): container finished" podID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerID="96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038" exitCode=0 Jul 15 19:03:04.561197 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561190 2561 generic.go:356] "Generic (PLEG): container finished" podID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerID="a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32" exitCode=0 Jul 15 19:03:04.561197 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561198 2561 generic.go:356] "Generic (PLEG): container finished" podID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerID="903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7" exitCode=0 Jul 15 19:03:04.561197 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561203 2561 generic.go:356] "Generic (PLEG): container finished" podID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerID="345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63" exitCode=0 Jul 15 19:03:04.561197 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561208 2561 generic.go:356] "Generic (PLEG): container finished" podID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerID="ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0" exitCode=0 Jul 15 19:03:04.561701 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561234 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038"} Jul 15 19:03:04.561701 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561297 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32"} Jul 15 19:03:04.561701 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561323 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7"} Jul 15 19:03:04.561701 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561350 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63"} Jul 15 19:03:04.561701 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:04.561376 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0"} Jul 15 19:03:05.010510 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.010485 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.126325 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126024 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.126325 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126306 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126666 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-tls-assets\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126733 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126772 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-web\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126805 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-out\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126841 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-cluster-tls-config\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126883 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-metrics-client-ca\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126939 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-volume\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.126978 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbln8\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-kube-api-access-fbln8\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.127020 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-main-db\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.127050 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-web-config\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.127115 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-trusted-ca-bundle\") pod \"59fc6268-8e25-45cc-bed2-92976c5e95c4\" (UID: \"59fc6268-8e25-45cc-bed2-92976c5e95c4\") " Jul 15 19:03:05.128208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.127903 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:05.129534 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.129251 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:05.131001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.130738 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:03:05.132648 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.132574 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:03:05.133209 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.132838 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:05.133209 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.133108 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:05.134890 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.134867 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:05.135196 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.135165 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-out" (OuterVolumeSpecName: "config-out") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:03:05.135578 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.135554 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-kube-api-access-fbln8" (OuterVolumeSpecName: "kube-api-access-fbln8") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "kube-api-access-fbln8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:03:05.135683 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.135664 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-volume" (OuterVolumeSpecName: "config-volume") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:05.136639 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.136586 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:05.140391 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.140369 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:05.146432 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.146410 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-web-config" (OuterVolumeSpecName: "web-config") pod "59fc6268-8e25-45cc-bed2-92976c5e95c4" (UID: "59fc6268-8e25-45cc-bed2-92976c5e95c4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:05.228624 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228577 2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228624 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228634 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228645 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-main-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228655 2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-tls-assets\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228665 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228674 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228684 2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-out\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228693 2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-cluster-tls-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228701 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59fc6268-8e25-45cc-bed2-92976c5e95c4-metrics-client-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228709 2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-config-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228717 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbln8\" (UniqueName: \"kubernetes.io/projected/59fc6268-8e25-45cc-bed2-92976c5e95c4-kube-api-access-fbln8\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228726 2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59fc6268-8e25-45cc-bed2-92976c5e95c4-alertmanager-main-db\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.228836 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.228734 2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59fc6268-8e25-45cc-bed2-92976c5e95c4-web-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:05.423353 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.423259 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4hlkp" Jul 15 19:03:05.568916 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.568881 2561 generic.go:356] "Generic (PLEG): container finished" podID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerID="3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878" exitCode=0 Jul 15 19:03:05.569322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.568955 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878"} Jul 15 19:03:05.569322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.569005 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.569322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.569024 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59fc6268-8e25-45cc-bed2-92976c5e95c4","Type":"ContainerDied","Data":"af3824477d3c6b44a7fd996b5e1ba997d1d1265de4f22a04b5629311475b4b6d"} Jul 15 19:03:05.569322 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.569068 2561 scope.go:117] "RemoveContainer" containerID="96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038" Jul 15 19:03:05.577197 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.577176 2561 scope.go:117] "RemoveContainer" containerID="a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32" Jul 15 19:03:05.584960 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.584941 2561 scope.go:117] "RemoveContainer" containerID="903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7" Jul 15 19:03:05.593100 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.593077 2561 scope.go:117] "RemoveContainer" containerID="3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878" Jul 15 19:03:05.593763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.593741 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:03:05.600856 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.600821 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:03:05.601714 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.601696 2561 scope.go:117] "RemoveContainer" containerID="345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63" Jul 15 19:03:05.608773 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.608752 2561 scope.go:117] "RemoveContainer" containerID="ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0" Jul 15 19:03:05.616037 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.616019 2561 scope.go:117] "RemoveContainer" containerID="2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a" Jul 15 19:03:05.623364 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.623346 2561 scope.go:117] "RemoveContainer" containerID="96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038" Jul 15 19:03:05.623679 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:05.623658 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038\": container with ID starting with 96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038 not found: ID does not exist" containerID="96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038" Jul 15 19:03:05.623740 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.623689 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038"} err="failed to get container status \"96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038\": rpc error: code = NotFound desc = could not find container \"96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038\": container with ID starting with 96ade9d5886d16ee323e9dbe02706d96faa68fe7f76eb9b028e76db1c238f038 not found: ID does not exist" Jul 15 19:03:05.623740 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.623709 2561 scope.go:117] "RemoveContainer" containerID="a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32" Jul 15 19:03:05.623934 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:05.623918 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32\": container with ID starting with a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32 not found: ID does not exist" containerID="a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32" Jul 15 19:03:05.623975 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.623940 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32"} err="failed to get container status \"a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32\": rpc error: code = NotFound desc = could not find container \"a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32\": container with ID starting with a3dccc252c47c515d231484a0887651733a500543860aeb7d971b22c115f8e32 not found: ID does not exist" Jul 15 19:03:05.623975 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.623954 2561 scope.go:117] "RemoveContainer" containerID="903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7" Jul 15 19:03:05.624154 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:05.624140 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7\": container with ID starting with 903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7 not found: ID does not exist" containerID="903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7" Jul 15 19:03:05.624193 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.624156 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7"} err="failed to get container status \"903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7\": rpc error: code = NotFound desc = could not find container \"903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7\": container with ID starting with 903e1197d6872ad0843cbcd50723fc5c620cacc898fc2e99483fdb6af8d022b7 not found: ID does not exist" Jul 15 19:03:05.624193 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.624168 2561 scope.go:117] "RemoveContainer" containerID="3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878" Jul 15 19:03:05.624508 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:05.624467 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878\": container with ID starting with 3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878 not found: ID does not exist" containerID="3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878" Jul 15 19:03:05.624843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.624504 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878"} err="failed to get container status \"3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878\": rpc error: code = NotFound desc = could not find container \"3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878\": container with ID starting with 3bf88a72274eb51404422b41ec1dc155b31ef5dee8432a77b94f19e385470878 not found: ID does not exist" Jul 15 19:03:05.624843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.624531 2561 scope.go:117] "RemoveContainer" containerID="345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63" Jul 15 19:03:05.625048 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:05.624904 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63\": container with ID starting with 345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63 not found: ID does not exist" containerID="345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63" Jul 15 19:03:05.625048 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.624931 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63"} err="failed to get container status \"345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63\": rpc error: code = NotFound desc = could not find container \"345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63\": container with ID starting with 345909da6fb365d48fc7de524b62ac858a180501354c9df30377c08d1930ee63 not found: ID does not exist" Jul 15 19:03:05.625048 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.624951 2561 scope.go:117] "RemoveContainer" containerID="ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0" Jul 15 19:03:05.625221 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:05.625194 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0\": container with ID starting with ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0 not found: ID does not exist" containerID="ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0" Jul 15 19:03:05.625263 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.625230 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0"} err="failed to get container status \"ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0\": rpc error: code = NotFound desc = could not find container \"ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0\": container with ID starting with ddeac7110057dc45e9e47353c290de18a9921993a68f4af24258704ab862fdc0 not found: ID does not exist" Jul 15 19:03:05.625263 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.625251 2561 scope.go:117] "RemoveContainer" containerID="2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a" Jul 15 19:03:05.625674 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:05.625516 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a\": container with ID starting with 2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a not found: ID does not exist" containerID="2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a" Jul 15 19:03:05.625674 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.625544 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a"} err="failed to get container status \"2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a\": rpc error: code = NotFound desc = could not find container \"2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a\": container with ID starting with 2368cbf632ba06da4e44abd393bfe76df6ea0506e523d91d8cd5059e3704154a not found: ID does not exist" Jul 15 19:03:05.626760 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.626741 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:03:05.627059 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627040 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a509a1-c7c5-4b61-a37f-5766f1daddc0" containerName="console" Jul 15 19:03:05.627059 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627058 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a509a1-c7c5-4b61-a37f-5766f1daddc0" containerName="console" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627070 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7815e9e9-2b77-4df2-be39-15b2378fb29b" containerName="console" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627076 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7815e9e9-2b77-4df2-be39-15b2378fb29b" containerName="console" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627088 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="init-config-reloader" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627096 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="init-config-reloader" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627113 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="alertmanager" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627122 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="alertmanager" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627133 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627139 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627148 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-metric" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627156 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-metric" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627164 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="prom-label-proxy" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627172 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="prom-label-proxy" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627183 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="config-reloader" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627192 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="config-reloader" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627204 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-web" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627212 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-web" Jul 15 19:03:05.627246 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627287 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy" Jul 15 19:03:05.627862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627301 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="config-reloader" Jul 15 19:03:05.627862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627312 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a509a1-c7c5-4b61-a37f-5766f1daddc0" containerName="console" Jul 15 19:03:05.627862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627320 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="prom-label-proxy" Jul 15 19:03:05.627862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627328 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-metric" Jul 15 19:03:05.627862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627337 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="alertmanager" Jul 15 19:03:05.627862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627353 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" containerName="kube-rbac-proxy-web" Jul 15 19:03:05.627862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.627363 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7815e9e9-2b77-4df2-be39-15b2378fb29b" containerName="console" Jul 15 19:03:05.630948 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.630932 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.634185 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.634059 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 19:03:05.634566 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.634387 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 19:03:05.634896 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.634785 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 19:03:05.635028 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.634893 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 19:03:05.635099 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.635076 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 19:03:05.635099 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.635079 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 19:03:05.635251 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.635159 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 19:03:05.635706 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.635688 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 19:03:05.635771 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.635717 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k5s6j\"" Jul 15 19:03:05.640868 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.640850 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 19:03:05.646817 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.646794 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:03:05.732478 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732478 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732441 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732478 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj979\" (UniqueName: \"kubernetes.io/projected/cee02323-f6b2-4f61-91e2-ae47058a9ffe-kube-api-access-lj979\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732536 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-web-config\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cee02323-f6b2-4f61-91e2-ae47058a9ffe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732630 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cee02323-f6b2-4f61-91e2-ae47058a9ffe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732709 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cee02323-f6b2-4f61-91e2-ae47058a9ffe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732754 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cee02323-f6b2-4f61-91e2-ae47058a9ffe-config-out\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.732797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732794 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.733075 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-config-volume\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.733075 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.732859 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee02323-f6b2-4f61-91e2-ae47058a9ffe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833615 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833577 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cee02323-f6b2-4f61-91e2-ae47058a9ffe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833738 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833634 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cee02323-f6b2-4f61-91e2-ae47058a9ffe-config-out\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833738 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833738 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833689 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-config-volume\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833738 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833707 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee02323-f6b2-4f61-91e2-ae47058a9ffe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833738 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833729 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833983 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833757 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833983 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833780 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj979\" (UniqueName: \"kubernetes.io/projected/cee02323-f6b2-4f61-91e2-ae47058a9ffe-kube-api-access-lj979\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833983 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833810 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833983 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833853 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833983 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-web-config\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833983 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833907 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cee02323-f6b2-4f61-91e2-ae47058a9ffe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.833983 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.833929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cee02323-f6b2-4f61-91e2-ae47058a9ffe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.834664 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.834635 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cee02323-f6b2-4f61-91e2-ae47058a9ffe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.834755 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.834686 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee02323-f6b2-4f61-91e2-ae47058a9ffe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.834978 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.834952 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cee02323-f6b2-4f61-91e2-ae47058a9ffe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.837017 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.836859 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.837132 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.837026 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cee02323-f6b2-4f61-91e2-ae47058a9ffe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.837400 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.837311 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.837400 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.837335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.837741 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.837700 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.837741 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.837724 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-config-volume\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.838027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.838003 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-web-config\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.838027 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.838018 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cee02323-f6b2-4f61-91e2-ae47058a9ffe-config-out\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.839327 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.839310 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cee02323-f6b2-4f61-91e2-ae47058a9ffe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.855664 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.855634 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj979\" (UniqueName: \"kubernetes.io/projected/cee02323-f6b2-4f61-91e2-ae47058a9ffe-kube-api-access-lj979\") pod \"alertmanager-main-0\" (UID: \"cee02323-f6b2-4f61-91e2-ae47058a9ffe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:05.941488 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:05.941449 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 19:03:06.080764 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:06.080728 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 19:03:06.084396 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:03:06.084366 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee02323_f6b2_4f61_91e2_ae47058a9ffe.slice/crio-56dcaabfd29747d00ece1fb5c09f7570fbe9301518e790b96818b4b59eeaac79 WatchSource:0}: Error finding container 56dcaabfd29747d00ece1fb5c09f7570fbe9301518e790b96818b4b59eeaac79: Status 404 returned error can't find the container with id 56dcaabfd29747d00ece1fb5c09f7570fbe9301518e790b96818b4b59eeaac79 Jul 15 19:03:06.574553 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:06.574512 2561 generic.go:356] "Generic (PLEG): container finished" podID="cee02323-f6b2-4f61-91e2-ae47058a9ffe" containerID="4f7796af347f343b250c88b64bf9ef6d4f1b28a8a613fc04242de7db3f95dea5" exitCode=0 Jul 15 19:03:06.575051 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:06.574599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerDied","Data":"4f7796af347f343b250c88b64bf9ef6d4f1b28a8a613fc04242de7db3f95dea5"} Jul 15 19:03:06.575051 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:06.574700 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerStarted","Data":"56dcaabfd29747d00ece1fb5c09f7570fbe9301518e790b96818b4b59eeaac79"} Jul 15 19:03:06.931703 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:06.931669 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59fc6268-8e25-45cc-bed2-92976c5e95c4" path="/var/lib/kubelet/pods/59fc6268-8e25-45cc-bed2-92976c5e95c4/volumes" Jul 15 19:03:07.582718 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:07.582688 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerStarted","Data":"4c61184e49e328bae7d3e4139d1ca2a956497f76f84ab481866eef1c5b5543ce"} Jul 15 19:03:07.583118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:07.582752 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerStarted","Data":"8783fb4540678d3ffb4f8d0d0774a210037afeae022d51d12dd546b4fc69d671"} Jul 15 19:03:07.583118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:07.582780 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerStarted","Data":"a0e8efbf524f3b2f3dcea0102794c57e3b4f97d9c7b51a4ed105c8222d95cf32"} Jul 15 19:03:07.583118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:07.582806 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerStarted","Data":"a3d101894c36be4aa9625c3ca33d131a036e38d34c8db48ae21612cdf1b64b67"} Jul 15 19:03:07.583118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:07.582828 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerStarted","Data":"22448ae2e5575328e5e0aaaed911da7aab5fd652fcc54d9c0960d9acda8e058a"} Jul 15 19:03:07.583118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:07.582849 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cee02323-f6b2-4f61-91e2-ae47058a9ffe","Type":"ContainerStarted","Data":"45e71b84a52a29ab86a3d00fa7db64fb15ac0edd44018f3b4d4ec0a2549df76d"} Jul 15 19:03:08.054944 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.054868 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.038836763 podStartE2EDuration="3.054845009s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.008246ms" isStatefulPod=false podCreationTimestamp="2026-07-15 19:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:03:07.622781537 +0000 UTC m=+103.456283102" watchObservedRunningTime="2026-07-15 19:03:08.054845009 +0000 UTC m=+103.888346562" Jul 15 19:03:08.055467 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.055448 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:03:08.055780 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.055753 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="prometheus" containerID="cri-o://1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc" gracePeriod=600 Jul 15 19:03:08.055908 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.055775 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy" containerID="cri-o://82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca" gracePeriod=600 Jul 15 19:03:08.055908 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.055805 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="thanos-sidecar" containerID="cri-o://b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557" gracePeriod=600 Jul 15 19:03:08.055908 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.055841 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-web" containerID="cri-o://d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5" gracePeriod=600 Jul 15 19:03:08.055908 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.055808 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="config-reloader" containerID="cri-o://1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27" gracePeriod=600 Jul 15 19:03:08.056149 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.055975 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c" gracePeriod=600 Jul 15 19:03:08.591089 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591054 2561 generic.go:356] "Generic (PLEG): container finished" podID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerID="980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c" exitCode=0 Jul 15 19:03:08.591089 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591080 2561 generic.go:356] "Generic (PLEG): container finished" podID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerID="82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca" exitCode=0 Jul 15 19:03:08.591089 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591089 2561 generic.go:356] "Generic (PLEG): container finished" podID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerID="b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557" exitCode=0 Jul 15 19:03:08.591089 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591096 2561 generic.go:356] "Generic (PLEG): container finished" podID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerID="1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27" exitCode=0 Jul 15 19:03:08.591852 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591103 2561 generic.go:356] "Generic (PLEG): container finished" podID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerID="1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc" exitCode=0 Jul 15 19:03:08.591852 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591122 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c"} Jul 15 19:03:08.591852 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591185 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca"} Jul 15 19:03:08.591852 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591212 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557"} Jul 15 19:03:08.591852 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591238 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27"} Jul 15 19:03:08.591852 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:08.591264 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc"} Jul 15 19:03:09.344559 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.344528 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.470896 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470796 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-metrics-client-certs\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.470896 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470835 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-tls\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.470896 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470863 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-thanos-prometheus-http-client-file\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470907 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-kubelet-serving-ca-bundle\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470931 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-trusted-ca-bundle\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470951 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-kube-rbac-proxy\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470972 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-serving-certs-ca-bundle\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.470999 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471025 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-config-out\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471057 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471106 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-1\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471135 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-db\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471158 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-web-config\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471185 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-grpc-tls\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471208 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-tls-assets\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471229 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-config\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471253 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-2\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471289 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t77x\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-kube-api-access-8t77x\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471316 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-0\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471366 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-metrics-client-ca\") pod \"40c0c8f8-86bf-4172-855b-9c57495d931e\" (UID: \"40c0c8f8-86bf-4172-855b-9c57495d931e\") " Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471383 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:09.471711 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471667 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.472150 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471918 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:09.472150 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.471995 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:09.472150 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.472037 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:09.472710 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.472645 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:09.473192 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.473110 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:09.473332 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.473305 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:03:09.474749 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.474707 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:03:09.474857 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.474750 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.474857 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.474778 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-config-out" (OuterVolumeSpecName: "config-out") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:03:09.475416 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.475375 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.475521 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.475472 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.475813 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.475780 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.476103 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.476067 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-config" (OuterVolumeSpecName: "config") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.476103 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.476091 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-kube-api-access-8t77x" (OuterVolumeSpecName: "kube-api-access-8t77x") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "kube-api-access-8t77x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:03:09.476281 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.476141 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.476281 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.476214 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:03:09.476368 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.476350 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.477124 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.477101 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.487910 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.487879 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-web-config" (OuterVolumeSpecName: "web-config") pod "40c0c8f8-86bf-4172-855b-9c57495d931e" (UID: "40c0c8f8-86bf-4172-855b-9c57495d931e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:03:09.573019 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.572982 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573019 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573015 2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-kube-rbac-proxy\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573019 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573026 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573035 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573045 2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-config-out\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573054 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573064 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573072 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-db\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573081 2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-web-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573089 2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-grpc-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573097 2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-tls-assets\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573104 2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-config\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573111 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573120 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t77x\" (UniqueName: \"kubernetes.io/projected/40c0c8f8-86bf-4172-855b-9c57495d931e-kube-api-access-8t77x\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573130 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573138 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40c0c8f8-86bf-4172-855b-9c57495d931e-configmap-metrics-client-ca\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573146 2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-metrics-client-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573154 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.573255 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.573162 2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40c0c8f8-86bf-4172-855b-9c57495d931e-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:03:09.599061 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.599030 2561 generic.go:356] "Generic (PLEG): container finished" podID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerID="d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5" exitCode=0 Jul 15 19:03:09.599548 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.599150 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5"} Jul 15 19:03:09.599548 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.599175 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.599548 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.599237 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40c0c8f8-86bf-4172-855b-9c57495d931e","Type":"ContainerDied","Data":"a2c9849aaf6450e2125fb44c7ba113b1a7d0729a47c0568c1c9034fa92f9c731"} Jul 15 19:03:09.599548 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.599287 2561 scope.go:117] "RemoveContainer" containerID="980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c" Jul 15 19:03:09.612306 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.612263 2561 scope.go:117] "RemoveContainer" containerID="82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca" Jul 15 19:03:09.619827 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.619803 2561 scope.go:117] "RemoveContainer" containerID="d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5" Jul 15 19:03:09.627040 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.627014 2561 scope.go:117] "RemoveContainer" containerID="b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557" Jul 15 19:03:09.629959 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.629937 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:03:09.633368 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.633341 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:03:09.635515 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.635498 2561 scope.go:117] "RemoveContainer" containerID="1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27" Jul 15 19:03:09.642795 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.642771 2561 scope.go:117] "RemoveContainer" containerID="1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc" Jul 15 19:03:09.650421 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.650401 2561 scope.go:117] "RemoveContainer" containerID="cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079" Jul 15 19:03:09.657429 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.657410 2561 scope.go:117] "RemoveContainer" containerID="980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c" Jul 15 19:03:09.657745 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:09.657724 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c\": container with ID starting with 980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c not found: ID does not exist" containerID="980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c" Jul 15 19:03:09.657827 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.657759 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c"} err="failed to get container status \"980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c\": rpc error: code = NotFound desc = could not find container \"980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c\": container with ID starting with 980fa8e72d5bbbe142fade69eb336877a260e5054adfc5437b7973f3799a857c not found: ID does not exist" Jul 15 19:03:09.657827 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.657787 2561 scope.go:117] "RemoveContainer" containerID="82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca" Jul 15 19:03:09.658045 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:09.658025 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca\": container with ID starting with 82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca not found: ID does not exist" containerID="82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca" Jul 15 19:03:09.658087 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658052 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca"} err="failed to get container status \"82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca\": rpc error: code = NotFound desc = could not find container \"82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca\": container with ID starting with 82aaaac1398eccd1191e9146f9cf37eb0d1ee2aff779013edd7624d26761e4ca not found: ID does not exist" Jul 15 19:03:09.658087 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658069 2561 scope.go:117] "RemoveContainer" containerID="d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5" Jul 15 19:03:09.658302 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:09.658287 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5\": container with ID starting with d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5 not found: ID does not exist" containerID="d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5" Jul 15 19:03:09.658363 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658318 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5"} err="failed to get container status \"d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5\": rpc error: code = NotFound desc = could not find container \"d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5\": container with ID starting with d2a786ce8c1e2b0f4ed480725b9774539baa1f6a1ad046b33697f75a2a36dea5 not found: ID does not exist" Jul 15 19:03:09.658363 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658338 2561 scope.go:117] "RemoveContainer" containerID="b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557" Jul 15 19:03:09.658540 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:09.658524 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557\": container with ID starting with b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557 not found: ID does not exist" containerID="b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557" Jul 15 19:03:09.658580 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658542 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557"} err="failed to get container status \"b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557\": rpc error: code = NotFound desc = could not find container \"b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557\": container with ID starting with b8e9ee9ad8dc721f319edf95d03b1a1b0d477f8b24154873bda46b15ad829557 not found: ID does not exist" Jul 15 19:03:09.658580 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658556 2561 scope.go:117] "RemoveContainer" containerID="1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27" Jul 15 19:03:09.658787 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:09.658770 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27\": container with ID starting with 1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27 not found: ID does not exist" containerID="1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27" Jul 15 19:03:09.658826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658792 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27"} err="failed to get container status \"1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27\": rpc error: code = NotFound desc = could not find container \"1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27\": container with ID starting with 1c40d371565b6a67ce969bc170e297b1379747da73badc95fab8ae7877ef3c27 not found: ID does not exist" Jul 15 19:03:09.658826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.658807 2561 scope.go:117] "RemoveContainer" containerID="1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc" Jul 15 19:03:09.659010 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:09.658995 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc\": container with ID starting with 1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc not found: ID does not exist" containerID="1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc" Jul 15 19:03:09.659048 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.659013 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc"} err="failed to get container status \"1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc\": rpc error: code = NotFound desc = could not find container \"1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc\": container with ID starting with 1df070ea59713518034ecc77c7e88fd49bebf8f1341c51c88286eb7e2cda4acc not found: ID does not exist" Jul 15 19:03:09.659048 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.659025 2561 scope.go:117] "RemoveContainer" containerID="cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079" Jul 15 19:03:09.659221 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:03:09.659205 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079\": container with ID starting with cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079 not found: ID does not exist" containerID="cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079" Jul 15 19:03:09.659275 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.659228 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079"} err="failed to get container status \"cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079\": rpc error: code = NotFound desc = could not find container \"cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079\": container with ID starting with cf4079e00e351e53fa2af72fdd8c663671785bd9b97a3d26a1e3ea1e96961079 not found: ID does not exist" Jul 15 19:03:09.662385 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662363 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:03:09.662744 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662728 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="thanos-sidecar" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662746 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="thanos-sidecar" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662760 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662768 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662780 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-web" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662788 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-web" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662803 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="init-config-reloader" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662811 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="init-config-reloader" Jul 15 19:03:09.662826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662825 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="config-reloader" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662833 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="config-reloader" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662852 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-thanos" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662862 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-thanos" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662875 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="prometheus" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662884 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="prometheus" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662959 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="config-reloader" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662970 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="thanos-sidecar" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662980 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.662990 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-thanos" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.663000 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="kube-rbac-proxy-web" Jul 15 19:03:09.663206 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.663008 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" containerName="prometheus" Jul 15 19:03:09.699123 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.699090 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:03:09.699319 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.699247 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.701832 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.701808 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 19:03:09.702037 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.701839 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 19:03:09.702108 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.702046 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 19:03:09.702811 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.702796 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 19:03:09.703184 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703163 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 19:03:09.703184 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703172 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 19:03:09.703338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703164 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rhftj\"" Jul 15 19:03:09.703338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703263 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 19:03:09.703338 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703177 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fso41rpjh0g74\"" Jul 15 19:03:09.703581 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703561 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 19:03:09.703581 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703580 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 19:03:09.703757 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703633 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 19:03:09.703757 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703649 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 19:03:09.703757 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.703584 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 19:03:09.707045 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.707025 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 19:03:09.709527 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.709503 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 19:03:09.775447 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775410 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775447 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775446 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdb56\" (UniqueName: \"kubernetes.io/projected/173fec37-9c71-4522-bfac-f5b321c27fb6-kube-api-access-cdb56\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775699 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775469 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775699 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775699 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775581 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775699 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775637 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775699 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775665 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775699 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775689 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775901 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775724 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-config\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775901 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/173fec37-9c71-4522-bfac-f5b321c27fb6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775901 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775766 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775901 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775786 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/173fec37-9c71-4522-bfac-f5b321c27fb6-config-out\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775901 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775821 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775901 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775845 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-web-config\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.775901 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.776129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.776129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.776129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775967 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.776129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.775981 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.776129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.776014 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.876751 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876714 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.876751 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876757 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876775 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876792 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876822 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876859 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-config\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876891 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/173fec37-9c71-4522-bfac-f5b321c27fb6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876917 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/173fec37-9c71-4522-bfac-f5b321c27fb6-config-out\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876962 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.876983 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-web-config\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877011 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877057 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877091 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877115 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877141 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877189 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877214 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdb56\" (UniqueName: \"kubernetes.io/projected/173fec37-9c71-4522-bfac-f5b321c27fb6-kube-api-access-cdb56\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877249 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877496 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.877681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877636 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.878143 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.877815 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.878204 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.878180 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.878741 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.878459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.879003 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.878981 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.880360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.880034 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-config\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.880360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.880108 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/173fec37-9c71-4522-bfac-f5b321c27fb6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.880360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.880189 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.880360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.880198 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.880785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.880763 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.881188 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.881111 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/173fec37-9c71-4522-bfac-f5b321c27fb6-config-out\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.881440 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.881414 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.881521 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.881415 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/173fec37-9c71-4522-bfac-f5b321c27fb6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.882012 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.881993 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.882267 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.882250 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-web-config\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.882512 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.882494 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.882778 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.882757 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.882847 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.882761 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/173fec37-9c71-4522-bfac-f5b321c27fb6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:09.889048 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:09.889024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdb56\" (UniqueName: \"kubernetes.io/projected/173fec37-9c71-4522-bfac-f5b321c27fb6-kube-api-access-cdb56\") pod \"prometheus-k8s-0\" (UID: \"173fec37-9c71-4522-bfac-f5b321c27fb6\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:10.011960 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:10.011920 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:03:10.189966 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:10.189783 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 19:03:10.192088 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:03:10.192055 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173fec37_9c71_4522_bfac_f5b321c27fb6.slice/crio-d1a3d25a7da61cf8d3ab847e7cd61c3f89a240a247aba54c1eaf9b59d301223b WatchSource:0}: Error finding container d1a3d25a7da61cf8d3ab847e7cd61c3f89a240a247aba54c1eaf9b59d301223b: Status 404 returned error can't find the container with id d1a3d25a7da61cf8d3ab847e7cd61c3f89a240a247aba54c1eaf9b59d301223b Jul 15 19:03:10.606915 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:10.606880 2561 generic.go:356] "Generic (PLEG): container finished" podID="173fec37-9c71-4522-bfac-f5b321c27fb6" containerID="bc5bf217295090ac145d8f4b6b2953e85a4da7f5f5b895d67b4b167eaddc0d16" exitCode=0 Jul 15 19:03:10.607295 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:10.606966 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerDied","Data":"bc5bf217295090ac145d8f4b6b2953e85a4da7f5f5b895d67b4b167eaddc0d16"} Jul 15 19:03:10.607295 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:10.607027 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerStarted","Data":"d1a3d25a7da61cf8d3ab847e7cd61c3f89a240a247aba54c1eaf9b59d301223b"} Jul 15 19:03:10.933066 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:10.933029 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c0c8f8-86bf-4172-855b-9c57495d931e" path="/var/lib/kubelet/pods/40c0c8f8-86bf-4172-855b-9c57495d931e/volumes" Jul 15 19:03:11.614563 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:11.614524 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerStarted","Data":"ec04a8d3f4cc4cdaacc05a5eeecd290d21227b86855a8c73926496258bbeb745"} Jul 15 19:03:11.615003 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:11.614637 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerStarted","Data":"dc75ce5dba4ab1eaa1411e32a761470ff498847f547857e67f7a61aa9d6af43b"} Jul 15 19:03:11.615003 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:11.614683 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerStarted","Data":"a49f0785e6227485d14c3597e4e84a84640baa04476d6621e94715569cd05fb1"} Jul 15 19:03:11.615003 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:11.614729 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerStarted","Data":"a18e8787deef8486ba3ce3c5fbe330e7e1584ff24e7b15328546759dc724d05a"} Jul 15 19:03:11.615003 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:11.614764 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerStarted","Data":"d0aee924fe6b02d80611c5d43b6233bbc611f5918cb65b50abdb32c7269d31b9"} Jul 15 19:03:11.615003 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:11.614800 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"173fec37-9c71-4522-bfac-f5b321c27fb6","Type":"ContainerStarted","Data":"114cbc83ad49e04f42ec2ada028a62df80977467fcaa087c3aae9acb9fd65350"} Jul 15 19:03:11.648915 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:11.648847 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.629247275 podStartE2EDuration="2.648821053s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.573778ms" isStatefulPod=false podCreationTimestamp="2026-07-15 19:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:03:11.647910622 +0000 UTC m=+107.481412211" watchObservedRunningTime="2026-07-15 19:03:11.648821053 +0000 UTC m=+107.482322606" Jul 15 19:03:15.012519 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:03:15.012472 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:04:10.012534 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:10.012495 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:04:10.028392 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:10.028363 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:04:10.866766 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:10.866738 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 19:04:48.822536 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.822505 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kdr4"] Jul 15 19:04:48.825826 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.825808 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:48.828576 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.828541 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sbjtz\"" Jul 15 19:04:48.828723 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.828703 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 19:04:48.829525 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.829508 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 19:04:48.853235 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.853201 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kdr4"] Jul 15 19:04:48.945387 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.945349 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83a23feb-d88c-4efe-a047-710cd9c6f93b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kdr4\" (UID: \"83a23feb-d88c-4efe-a047-710cd9c6f93b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:48.945580 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:48.945394 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbpz\" (UniqueName: \"kubernetes.io/projected/83a23feb-d88c-4efe-a047-710cd9c6f93b-kube-api-access-pfbpz\") pod \"cert-manager-webhook-6486ffd997-4kdr4\" (UID: \"83a23feb-d88c-4efe-a047-710cd9c6f93b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:49.045977 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:49.045941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83a23feb-d88c-4efe-a047-710cd9c6f93b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kdr4\" (UID: \"83a23feb-d88c-4efe-a047-710cd9c6f93b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:49.046176 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:49.045983 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbpz\" (UniqueName: \"kubernetes.io/projected/83a23feb-d88c-4efe-a047-710cd9c6f93b-kube-api-access-pfbpz\") pod \"cert-manager-webhook-6486ffd997-4kdr4\" (UID: \"83a23feb-d88c-4efe-a047-710cd9c6f93b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:49.059764 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:49.059734 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbpz\" (UniqueName: \"kubernetes.io/projected/83a23feb-d88c-4efe-a047-710cd9c6f93b-kube-api-access-pfbpz\") pod \"cert-manager-webhook-6486ffd997-4kdr4\" (UID: \"83a23feb-d88c-4efe-a047-710cd9c6f93b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:49.060646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:49.060625 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83a23feb-d88c-4efe-a047-710cd9c6f93b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kdr4\" (UID: \"83a23feb-d88c-4efe-a047-710cd9c6f93b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:49.145083 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:49.144991 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:49.294458 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:49.294434 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kdr4"] Jul 15 19:04:49.296005 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:04:49.295979 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83a23feb_d88c_4efe_a047_710cd9c6f93b.slice/crio-637db534005c85231818c54bbdf8b5683612f96066d37bbaa15d7b7d84ea819d WatchSource:0}: Error finding container 637db534005c85231818c54bbdf8b5683612f96066d37bbaa15d7b7d84ea819d: Status 404 returned error can't find the container with id 637db534005c85231818c54bbdf8b5683612f96066d37bbaa15d7b7d84ea819d Jul 15 19:04:50.001490 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.001451 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" event={"ID":"83a23feb-d88c-4efe-a047-710cd9c6f93b","Type":"ContainerStarted","Data":"637db534005c85231818c54bbdf8b5683612f96066d37bbaa15d7b7d84ea819d"} Jul 15 19:04:50.442438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.442403 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lz65m"] Jul 15 19:04:50.446046 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.446018 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.448967 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.448941 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7xpp2\"" Jul 15 19:04:50.458978 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.458943 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lz65m"] Jul 15 19:04:50.560363 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.560326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swf9\" (UniqueName: \"kubernetes.io/projected/ae4b4df5-e77e-427a-af46-344945b48512-kube-api-access-2swf9\") pod \"cert-manager-cainjector-559cdd8c87-lz65m\" (UID: \"ae4b4df5-e77e-427a-af46-344945b48512\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.560567 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.560474 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4b4df5-e77e-427a-af46-344945b48512-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lz65m\" (UID: \"ae4b4df5-e77e-427a-af46-344945b48512\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.661476 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.661434 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swf9\" (UniqueName: \"kubernetes.io/projected/ae4b4df5-e77e-427a-af46-344945b48512-kube-api-access-2swf9\") pod \"cert-manager-cainjector-559cdd8c87-lz65m\" (UID: \"ae4b4df5-e77e-427a-af46-344945b48512\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.661696 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.661546 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4b4df5-e77e-427a-af46-344945b48512-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lz65m\" (UID: \"ae4b4df5-e77e-427a-af46-344945b48512\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.672140 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.672081 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4b4df5-e77e-427a-af46-344945b48512-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lz65m\" (UID: \"ae4b4df5-e77e-427a-af46-344945b48512\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.674506 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.674455 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swf9\" (UniqueName: \"kubernetes.io/projected/ae4b4df5-e77e-427a-af46-344945b48512-kube-api-access-2swf9\") pod \"cert-manager-cainjector-559cdd8c87-lz65m\" (UID: \"ae4b4df5-e77e-427a-af46-344945b48512\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.759131 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.759047 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" Jul 15 19:04:50.935423 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:50.935389 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lz65m"] Jul 15 19:04:50.939269 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:04:50.939232 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae4b4df5_e77e_427a_af46_344945b48512.slice/crio-f94938ff536e943a466f12bc845f41d9a183b01d98fd5dea9623808775c56b52 WatchSource:0}: Error finding container f94938ff536e943a466f12bc845f41d9a183b01d98fd5dea9623808775c56b52: Status 404 returned error can't find the container with id f94938ff536e943a466f12bc845f41d9a183b01d98fd5dea9623808775c56b52 Jul 15 19:04:51.009028 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:51.008986 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" event={"ID":"ae4b4df5-e77e-427a-af46-344945b48512","Type":"ContainerStarted","Data":"f94938ff536e943a466f12bc845f41d9a183b01d98fd5dea9623808775c56b52"} Jul 15 19:04:53.021788 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:53.021752 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" event={"ID":"83a23feb-d88c-4efe-a047-710cd9c6f93b","Type":"ContainerStarted","Data":"210eac6a821335d528429522c2fa53b778aa2e66a7a328f149fd6e1b82012978"} Jul 15 19:04:53.022266 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:53.021904 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:04:53.023310 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:53.023287 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" event={"ID":"ae4b4df5-e77e-427a-af46-344945b48512","Type":"ContainerStarted","Data":"a72c5bfb1de0d9c877bb824df100d956258860f34fed2098911d2e335aa954a5"} Jul 15 19:04:53.042728 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:53.042673 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" podStartSLOduration=1.645030437 podStartE2EDuration="5.042644303s" totalImagesPullingTime="3.397613866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:04:53.040779454 +0000 UTC m=+208.874281031" watchObservedRunningTime="2026-07-15 19:04:53.042644303 +0000 UTC m=+208.876145855" Jul 15 19:04:53.058351 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:53.058287 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lz65m" podStartSLOduration=1.3042877449999999 podStartE2EDuration="3.058264833s" totalImagesPullingTime="1.753977088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:04:53.057347868 +0000 UTC m=+208.890849421" watchObservedRunningTime="2026-07-15 19:04:53.058264833 +0000 UTC m=+208.891766386" Jul 15 19:04:59.030773 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:04:59.030739 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4kdr4" Jul 15 19:05:35.279494 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.279459 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4"] Jul 15 19:05:35.284184 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.284164 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.289945 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.289918 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 19:05:35.291273 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.291244 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 19:05:35.291273 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.291256 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 19:05:35.292172 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.292149 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:05:35.292280 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.292153 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 19:05:35.292280 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.292157 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-khls9\"" Jul 15 19:05:35.324434 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.324395 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4"] Jul 15 19:05:35.373360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.373327 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a40ca77b-ed75-4221-a864-0cf4ae47d591-manager-config\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.373537 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.373370 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kj5j\" (UniqueName: \"kubernetes.io/projected/a40ca77b-ed75-4221-a864-0cf4ae47d591-kube-api-access-8kj5j\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.373537 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.373395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a40ca77b-ed75-4221-a864-0cf4ae47d591-metrics-cert\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.373537 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.373507 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40ca77b-ed75-4221-a864-0cf4ae47d591-cert\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.474021 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.473986 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kj5j\" (UniqueName: \"kubernetes.io/projected/a40ca77b-ed75-4221-a864-0cf4ae47d591-kube-api-access-8kj5j\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.474021 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.474026 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a40ca77b-ed75-4221-a864-0cf4ae47d591-metrics-cert\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.474291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.474071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40ca77b-ed75-4221-a864-0cf4ae47d591-cert\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.474291 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.474119 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a40ca77b-ed75-4221-a864-0cf4ae47d591-manager-config\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.474808 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.474786 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a40ca77b-ed75-4221-a864-0cf4ae47d591-manager-config\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.476573 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.476551 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a40ca77b-ed75-4221-a864-0cf4ae47d591-metrics-cert\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.476687 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.476630 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40ca77b-ed75-4221-a864-0cf4ae47d591-cert\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.487633 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.487592 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kj5j\" (UniqueName: \"kubernetes.io/projected/a40ca77b-ed75-4221-a864-0cf4ae47d591-kube-api-access-8kj5j\") pod \"lws-controller-manager-8659b4c98-nphq4\" (UID: \"a40ca77b-ed75-4221-a864-0cf4ae47d591\") " pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.594342 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.594255 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:35.747621 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:35.747573 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4"] Jul 15 19:05:35.753452 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:05:35.753425 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40ca77b_ed75_4221_a864_0cf4ae47d591.slice/crio-12730aa29732265c508c1755856c8fb40751747acf538e264895aea49e8e091b WatchSource:0}: Error finding container 12730aa29732265c508c1755856c8fb40751747acf538e264895aea49e8e091b: Status 404 returned error can't find the container with id 12730aa29732265c508c1755856c8fb40751747acf538e264895aea49e8e091b Jul 15 19:05:36.205511 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:36.205478 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" event={"ID":"a40ca77b-ed75-4221-a864-0cf4ae47d591","Type":"ContainerStarted","Data":"12730aa29732265c508c1755856c8fb40751747acf538e264895aea49e8e091b"} Jul 15 19:05:38.214955 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:38.214896 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" event={"ID":"a40ca77b-ed75-4221-a864-0cf4ae47d591","Type":"ContainerStarted","Data":"86f40de40f758392d9e6fbb9c25432bb82e6133670ce82ef41814e01bf224b2a"} Jul 15 19:05:38.215336 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:38.215048 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:38.245287 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:38.245184 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" podStartSLOduration=1.027486035 podStartE2EDuration="3.245167316s" totalImagesPullingTime="2.217681281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:05:38.242875018 +0000 UTC m=+254.076376570" watchObservedRunningTime="2026-07-15 19:05:38.245167316 +0000 UTC m=+254.078668867" Jul 15 19:05:39.012625 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.012578 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz"] Jul 15 19:05:39.015234 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.015213 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.019124 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.019106 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 19:05:39.019438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.019416 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 19:05:39.019527 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.019427 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 19:05:39.019527 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.019454 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 19:05:39.019527 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.019455 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rf8qx\"" Jul 15 19:05:39.056362 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.056333 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz"] Jul 15 19:05:39.109005 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.108971 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a310d8f-211f-4c02-9b96-bb7704cd08f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.109005 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.109005 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslkz\" (UniqueName: \"kubernetes.io/projected/4a310d8f-211f-4c02-9b96-bb7704cd08f8-kube-api-access-bslkz\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.109221 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.109084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a310d8f-211f-4c02-9b96-bb7704cd08f8-tmp\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.109221 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.109110 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a310d8f-211f-4c02-9b96-bb7704cd08f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.109221 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.109147 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a310d8f-211f-4c02-9b96-bb7704cd08f8-manifests\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.209753 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.209706 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a310d8f-211f-4c02-9b96-bb7704cd08f8-manifests\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.209942 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.209794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a310d8f-211f-4c02-9b96-bb7704cd08f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.209942 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.209814 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslkz\" (UniqueName: \"kubernetes.io/projected/4a310d8f-211f-4c02-9b96-bb7704cd08f8-kube-api-access-bslkz\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.209942 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.209853 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a310d8f-211f-4c02-9b96-bb7704cd08f8-tmp\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.209942 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.209869 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a310d8f-211f-4c02-9b96-bb7704cd08f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.210167 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.210099 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a310d8f-211f-4c02-9b96-bb7704cd08f8-manifests\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.212124 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.212098 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a310d8f-211f-4c02-9b96-bb7704cd08f8-tmp\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.212325 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.212307 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a310d8f-211f-4c02-9b96-bb7704cd08f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.212457 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.212436 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a310d8f-211f-4c02-9b96-bb7704cd08f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.218212 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.218189 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslkz\" (UniqueName: \"kubernetes.io/projected/4a310d8f-211f-4c02-9b96-bb7704cd08f8-kube-api-access-bslkz\") pod \"opendatahub-operator-controller-manager-55c89bff45-v65kz\" (UID: \"4a310d8f-211f-4c02-9b96-bb7704cd08f8\") " pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.326567 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.326522 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:39.469784 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:39.469757 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz"] Jul 15 19:05:39.472243 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:05:39.472216 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a310d8f_211f_4c02_9b96_bb7704cd08f8.slice/crio-d9c2762705a7cc2f6b63ba5fbd8a7ece3e05d4498773fffe45b9a78e392a021b WatchSource:0}: Error finding container d9c2762705a7cc2f6b63ba5fbd8a7ece3e05d4498773fffe45b9a78e392a021b: Status 404 returned error can't find the container with id d9c2762705a7cc2f6b63ba5fbd8a7ece3e05d4498773fffe45b9a78e392a021b Jul 15 19:05:40.224934 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:40.224899 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" event={"ID":"4a310d8f-211f-4c02-9b96-bb7704cd08f8","Type":"ContainerStarted","Data":"d9c2762705a7cc2f6b63ba5fbd8a7ece3e05d4498773fffe45b9a78e392a021b"} Jul 15 19:05:43.243989 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:43.243884 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" event={"ID":"4a310d8f-211f-4c02-9b96-bb7704cd08f8","Type":"ContainerStarted","Data":"85ec4385306a2e9edb52af2f26add5d735315bc3699dfd04eafb5e26efd0939f"} Jul 15 19:05:44.250051 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:44.250016 2561 generic.go:356] "Generic (PLEG): container finished" podID="4a310d8f-211f-4c02-9b96-bb7704cd08f8" containerID="85ec4385306a2e9edb52af2f26add5d735315bc3699dfd04eafb5e26efd0939f" exitCode=0 Jul 15 19:05:44.250524 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:44.250087 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" event={"ID":"4a310d8f-211f-4c02-9b96-bb7704cd08f8","Type":"ContainerDied","Data":"85ec4385306a2e9edb52af2f26add5d735315bc3699dfd04eafb5e26efd0939f"} Jul 15 19:05:45.257286 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:45.257188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" event={"ID":"4a310d8f-211f-4c02-9b96-bb7704cd08f8","Type":"ContainerStarted","Data":"606018d3564a2c41cbacc5cd01e943165486301cb3b5a6192426fb89dad3604f"} Jul 15 19:05:45.257677 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:45.257333 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:45.283951 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:45.283895 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" podStartSLOduration=2.155255356 podStartE2EDuration="7.283872203s" totalImagesPullingTime="4.031254845s" totalInitContainerRuntime="1.097362002s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:05:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:05:45.281539993 +0000 UTC m=+261.115041565" watchObservedRunningTime="2026-07-15 19:05:45.283872203 +0000 UTC m=+261.117373756" Jul 15 19:05:49.221274 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:49.221249 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8659b4c98-nphq4" Jul 15 19:05:56.263410 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:56.263365 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55c89bff45-v65kz" Jul 15 19:05:58.941825 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.941794 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt"] Jul 15 19:05:58.952532 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.952509 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:58.956658 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.956632 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 19:05:58.957357 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.957336 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 19:05:58.957519 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.957501 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7hxkf\"" Jul 15 19:05:58.957594 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.957505 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 19:05:58.964561 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.964538 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 19:05:58.965760 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.965739 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt"] Jul 15 19:05:58.995267 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.995238 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b270c2b6-c25f-47af-a5a7-64a38a31b22c-tls-certs\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:58.995532 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.995308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b270c2b6-c25f-47af-a5a7-64a38a31b22c-tmp\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:58.995532 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:58.995464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfj84\" (UniqueName: \"kubernetes.io/projected/b270c2b6-c25f-47af-a5a7-64a38a31b22c-kube-api-access-rfj84\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.096751 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.096716 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b270c2b6-c25f-47af-a5a7-64a38a31b22c-tls-certs\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.096751 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.096760 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b270c2b6-c25f-47af-a5a7-64a38a31b22c-tmp\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.097017 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.096808 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfj84\" (UniqueName: \"kubernetes.io/projected/b270c2b6-c25f-47af-a5a7-64a38a31b22c-kube-api-access-rfj84\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.099114 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.099076 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b270c2b6-c25f-47af-a5a7-64a38a31b22c-tmp\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.099347 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.099327 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b270c2b6-c25f-47af-a5a7-64a38a31b22c-tls-certs\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.119578 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.119544 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfj84\" (UniqueName: \"kubernetes.io/projected/b270c2b6-c25f-47af-a5a7-64a38a31b22c-kube-api-access-rfj84\") pod \"kube-auth-proxy-54f9c67c7d-8ljzt\" (UID: \"b270c2b6-c25f-47af-a5a7-64a38a31b22c\") " pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.262269 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.262186 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" Jul 15 19:05:59.410980 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:05:59.410951 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt"] Jul 15 19:05:59.413144 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:05:59.413114 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb270c2b6_c25f_47af_a5a7_64a38a31b22c.slice/crio-297be168e6c4eb143d71aa79e745a8b1db14621a0d68ce68550978edb8ce64f5 WatchSource:0}: Error finding container 297be168e6c4eb143d71aa79e745a8b1db14621a0d68ce68550978edb8ce64f5: Status 404 returned error can't find the container with id 297be168e6c4eb143d71aa79e745a8b1db14621a0d68ce68550978edb8ce64f5 Jul 15 19:06:00.331397 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:06:00.331348 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" event={"ID":"b270c2b6-c25f-47af-a5a7-64a38a31b22c","Type":"ContainerStarted","Data":"297be168e6c4eb143d71aa79e745a8b1db14621a0d68ce68550978edb8ce64f5"} Jul 15 19:06:03.347189 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:06:03.347156 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" event={"ID":"b270c2b6-c25f-47af-a5a7-64a38a31b22c","Type":"ContainerStarted","Data":"0a112415cbc5d2d744ce18d50adbdf17797be00c7212cf0ff7252436c0ee8fb9"} Jul 15 19:06:03.364571 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:06:03.364520 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54f9c67c7d-8ljzt" podStartSLOduration=1.760057059 podStartE2EDuration="5.364502634s" totalImagesPullingTime="3.604445575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:06:03.362954728 +0000 UTC m=+279.196456280" watchObservedRunningTime="2026-07-15 19:06:03.364502634 +0000 UTC m=+279.198004204" Jul 15 19:06:24.711324 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:06:24.711293 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:06:24.717123 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:06:24.717100 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:06:24.717759 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:06:24.717737 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 19:07:27.426208 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.425245 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-654f6498c5-x7scl"] Jul 15 19:07:27.431083 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.431057 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.433759 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.433735 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 19:07:27.433917 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.433883 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5mhbd\"" Jul 15 19:07:27.434722 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.434706 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 19:07:27.434722 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.434717 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 19:07:27.434885 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.434755 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 19:07:27.434885 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.434710 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 19:07:27.434885 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.434710 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 19:07:27.434885 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.434809 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 19:07:27.440510 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.440459 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 19:07:27.441544 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.441518 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654f6498c5-x7scl"] Jul 15 19:07:27.523033 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.522998 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-oauth-config\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.523033 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.523035 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-serving-cert\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.523263 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.523064 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-service-ca\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.523263 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.523143 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-trusted-ca-bundle\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.523263 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.523160 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-oauth-serving-cert\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.523263 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.523176 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhb5k\" (UniqueName: \"kubernetes.io/projected/795a48e6-d05c-4d12-bb36-60c04ab702b6-kube-api-access-vhb5k\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.523405 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.523266 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-config\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.624195 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.624163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-config\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.624376 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.624216 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-oauth-config\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.624376 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.624238 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-serving-cert\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.624376 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.624360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-service-ca\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.624560 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.624407 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-trusted-ca-bundle\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.624560 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.624430 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-oauth-serving-cert\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.624560 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.624449 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhb5k\" (UniqueName: \"kubernetes.io/projected/795a48e6-d05c-4d12-bb36-60c04ab702b6-kube-api-access-vhb5k\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.625088 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.625065 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-config\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.625165 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.625145 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-oauth-serving-cert\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.625212 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.625145 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-service-ca\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.625351 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.625329 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795a48e6-d05c-4d12-bb36-60c04ab702b6-trusted-ca-bundle\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.626713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.626691 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-oauth-config\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.626811 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.626794 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795a48e6-d05c-4d12-bb36-60c04ab702b6-console-serving-cert\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.636044 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.636019 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhb5k\" (UniqueName: \"kubernetes.io/projected/795a48e6-d05c-4d12-bb36-60c04ab702b6-kube-api-access-vhb5k\") pod \"console-654f6498c5-x7scl\" (UID: \"795a48e6-d05c-4d12-bb36-60c04ab702b6\") " pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.742121 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.742023 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:27.882182 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:27.882154 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654f6498c5-x7scl"] Jul 15 19:07:27.884830 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:07:27.884800 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795a48e6_d05c_4d12_bb36_60c04ab702b6.slice/crio-6191b2b7cfd7d41ee547316e25fb3c9f781e8003ff45c259bd83128b9b5811fc WatchSource:0}: Error finding container 6191b2b7cfd7d41ee547316e25fb3c9f781e8003ff45c259bd83128b9b5811fc: Status 404 returned error can't find the container with id 6191b2b7cfd7d41ee547316e25fb3c9f781e8003ff45c259bd83128b9b5811fc Jul 15 19:07:28.732677 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:28.732636 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654f6498c5-x7scl" event={"ID":"795a48e6-d05c-4d12-bb36-60c04ab702b6","Type":"ContainerStarted","Data":"5d795e65471b03f7c815bb3cdf5166f1705afe04e6ccbfe96d11e746c29bdf62"} Jul 15 19:07:28.733054 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:28.732697 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654f6498c5-x7scl" event={"ID":"795a48e6-d05c-4d12-bb36-60c04ab702b6","Type":"ContainerStarted","Data":"6191b2b7cfd7d41ee547316e25fb3c9f781e8003ff45c259bd83128b9b5811fc"} Jul 15 19:07:28.758670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:28.758596 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-654f6498c5-x7scl" podStartSLOduration=1.758577962 podStartE2EDuration="1.758577962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:07:28.756561526 +0000 UTC m=+364.590063079" watchObservedRunningTime="2026-07-15 19:07:28.758577962 +0000 UTC m=+364.592079513" Jul 15 19:07:37.742495 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:37.742459 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:37.743043 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:37.742543 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:37.747683 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:37.747650 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:37.780296 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:37.780267 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-654f6498c5-x7scl" Jul 15 19:07:53.867022 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.866988 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx"] Jul 15 19:07:53.870727 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.870707 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.874337 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.874316 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 19:07:53.874454 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.874319 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 19:07:53.875350 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.875323 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:07:53.875350 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.875349 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:07:53.875484 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.875383 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n6qmk\"" Jul 15 19:07:53.880532 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.880512 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.880654 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.880593 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.880720 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.880656 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rdc\" (UniqueName: \"kubernetes.io/projected/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-kube-api-access-f4rdc\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.880978 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.880961 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx"] Jul 15 19:07:53.981339 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.981303 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.981519 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.981398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.981519 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.981425 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rdc\" (UniqueName: \"kubernetes.io/projected/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-kube-api-access-f4rdc\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.982109 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.982082 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:53.983818 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:53.983798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:54.000441 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:54.000415 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rdc\" (UniqueName: \"kubernetes.io/projected/35e8b23c-5d3b-4e48-ad97-6ee7b4c68821-kube-api-access-f4rdc\") pod \"kuadrant-console-plugin-66b5774985-sspgx\" (UID: \"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:54.181478 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:54.181387 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" Jul 15 19:07:54.318333 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:54.318308 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx"] Jul 15 19:07:54.321013 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:07:54.320978 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35e8b23c_5d3b_4e48_ad97_6ee7b4c68821.slice/crio-f063ffe9dbe5320461cf047d4569aaf286511a50352a9d8c419a2fe3ae77eaf9 WatchSource:0}: Error finding container f063ffe9dbe5320461cf047d4569aaf286511a50352a9d8c419a2fe3ae77eaf9: Status 404 returned error can't find the container with id f063ffe9dbe5320461cf047d4569aaf286511a50352a9d8c419a2fe3ae77eaf9 Jul 15 19:07:54.322283 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:54.322267 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:07:54.852618 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:07:54.852575 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" event={"ID":"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821","Type":"ContainerStarted","Data":"f063ffe9dbe5320461cf047d4569aaf286511a50352a9d8c419a2fe3ae77eaf9"} Jul 15 19:08:04.525815 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.525780 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9"] Jul 15 19:08:04.533259 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.533230 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.536050 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.536017 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-klgdv\"" Jul 15 19:08:04.551843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.551813 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9"] Jul 15 19:08:04.588924 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.588883 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-728l9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.589110 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.588942 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmf4r\" (UniqueName: \"kubernetes.io/projected/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-kube-api-access-bmf4r\") pod \"kuadrant-operator-controller-manager-55467d9f7-728l9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.652397 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.652304 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9"] Jul 15 19:08:04.652703 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:08:04.652678 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-bmf4r], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" podUID="e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" Jul 15 19:08:04.689708 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.689671 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-728l9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.689905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.689720 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmf4r\" (UniqueName: \"kubernetes.io/projected/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-kube-api-access-bmf4r\") pod \"kuadrant-operator-controller-manager-55467d9f7-728l9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.690121 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.690088 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-728l9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.712189 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.712154 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmf4r\" (UniqueName: \"kubernetes.io/projected/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-kube-api-access-bmf4r\") pod \"kuadrant-operator-controller-manager-55467d9f7-728l9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.731324 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.731293 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9"] Jul 15 19:08:04.735141 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.735081 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9"] Jul 15 19:08:04.912069 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.911990 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.919120 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:08:04.919083 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-728l9\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.923070 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.923041 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.925780 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:08:04.925737 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-728l9\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.933982 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:08:04.933934 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-728l9\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:04.993627 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.993569 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-extensions-socket-volume\") pod \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " Jul 15 19:08:04.993812 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.993658 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmf4r\" (UniqueName: \"kubernetes.io/projected/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-kube-api-access-bmf4r\") pod \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\" (UID: \"e4e23fa1-f606-4122-8c0d-122aa1ffd3a9\") " Jul 15 19:08:04.993888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.993856 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" (UID: "e4e23fa1-f606-4122-8c0d-122aa1ffd3a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:08:04.993971 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.993954 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-extensions-socket-volume\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:08:04.996184 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:04.996157 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-kube-api-access-bmf4r" (OuterVolumeSpecName: "kube-api-access-bmf4r") pod "e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" (UID: "e4e23fa1-f606-4122-8c0d-122aa1ffd3a9"). InnerVolumeSpecName "kube-api-access-bmf4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:08:05.095192 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:05.095152 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmf4r\" (UniqueName: \"kubernetes.io/projected/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9-kube-api-access-bmf4r\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:08:05.917345 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:05.917312 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:05.926097 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:08:05.926054 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-728l9\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-728l9" Jul 15 19:08:06.933875 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:06.933821 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e23fa1-f606-4122-8c0d-122aa1ffd3a9" path="/var/lib/kubelet/pods/e4e23fa1-f606-4122-8c0d-122aa1ffd3a9/volumes" Jul 15 19:08:18.993445 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:18.993347 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" event={"ID":"35e8b23c-5d3b-4e48-ad97-6ee7b4c68821","Type":"ContainerStarted","Data":"b618c28528722d8c8abae4f667b218987babf7977f40f4e23dff53f593c6da1b"} Jul 15 19:08:19.010808 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:19.010743 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-sspgx" podStartSLOduration=1.6119743720000002 podStartE2EDuration="26.010722951s" totalImagesPullingTime="24.398748579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:19.010072593 +0000 UTC m=+414.843574207" watchObservedRunningTime="2026-07-15 19:08:19.010722951 +0000 UTC m=+414.844224507" Jul 15 19:08:42.899654 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:42.899593 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:08:42.944379 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:42.944340 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:42.946850 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:42.946827 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 19:08:42.950926 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:42.950904 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:08:42.951079 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:42.950951 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:08:43.053051 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.053011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6cnl\" (UniqueName: \"kubernetes.io/projected/a41079bb-6cfe-4886-a771-9bf0aff8600d-kube-api-access-l6cnl\") pod \"limitador-limitador-54d6d988d5-zgtjg\" (UID: \"a41079bb-6cfe-4886-a771-9bf0aff8600d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:43.053242 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.053144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a41079bb-6cfe-4886-a771-9bf0aff8600d-config-file\") pod \"limitador-limitador-54d6d988d5-zgtjg\" (UID: \"a41079bb-6cfe-4886-a771-9bf0aff8600d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:43.154438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.154344 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6cnl\" (UniqueName: \"kubernetes.io/projected/a41079bb-6cfe-4886-a771-9bf0aff8600d-kube-api-access-l6cnl\") pod \"limitador-limitador-54d6d988d5-zgtjg\" (UID: \"a41079bb-6cfe-4886-a771-9bf0aff8600d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:43.154438 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.154390 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a41079bb-6cfe-4886-a771-9bf0aff8600d-config-file\") pod \"limitador-limitador-54d6d988d5-zgtjg\" (UID: \"a41079bb-6cfe-4886-a771-9bf0aff8600d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:43.155128 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.155108 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a41079bb-6cfe-4886-a771-9bf0aff8600d-config-file\") pod \"limitador-limitador-54d6d988d5-zgtjg\" (UID: \"a41079bb-6cfe-4886-a771-9bf0aff8600d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:43.164242 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.164206 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6cnl\" (UniqueName: \"kubernetes.io/projected/a41079bb-6cfe-4886-a771-9bf0aff8600d-kube-api-access-l6cnl\") pod \"limitador-limitador-54d6d988d5-zgtjg\" (UID: \"a41079bb-6cfe-4886-a771-9bf0aff8600d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:43.255252 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.255218 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:43.389785 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:43.389758 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:08:43.391672 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:08:43.391645 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41079bb_6cfe_4886_a771_9bf0aff8600d.slice/crio-c79f3760ef2b425a8eebe75b52258bd05a8d3449e0258889851ae52b77031e6a WatchSource:0}: Error finding container c79f3760ef2b425a8eebe75b52258bd05a8d3449e0258889851ae52b77031e6a: Status 404 returned error can't find the container with id c79f3760ef2b425a8eebe75b52258bd05a8d3449e0258889851ae52b77031e6a Jul 15 19:08:44.112842 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:44.112805 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" event={"ID":"a41079bb-6cfe-4886-a771-9bf0aff8600d","Type":"ContainerStarted","Data":"c79f3760ef2b425a8eebe75b52258bd05a8d3449e0258889851ae52b77031e6a"} Jul 15 19:08:47.130702 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:47.130663 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" event={"ID":"a41079bb-6cfe-4886-a771-9bf0aff8600d","Type":"ContainerStarted","Data":"54cb2925b1767e35621f235343173597df91e6b7ffb523b8ade96ca01868d3a8"} Jul 15 19:08:47.131069 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:47.130824 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:08:47.158480 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:47.158422 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" podStartSLOduration=1.571325908 podStartE2EDuration="5.158404896s" totalImagesPullingTime="3.587078988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:08:47.156944603 +0000 UTC m=+442.990446188" watchObservedRunningTime="2026-07-15 19:08:47.158404896 +0000 UTC m=+442.991906447" Jul 15 19:08:58.136399 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:08:58.136362 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-zgtjg" Jul 15 19:10:00.227185 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:00.227145 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:10:21.295421 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:21.295339 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:10:24.996376 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:24.996339 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:10:35.233399 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.233371 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj"] Jul 15 19:10:35.236063 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.236040 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.238646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.238622 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 19:10:35.238646 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.238634 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 19:10:35.239421 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.239407 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6xgvb\"" Jul 15 19:10:35.239482 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.239435 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 19:10:35.249659 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.249634 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj"] Jul 15 19:10:35.310853 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.310816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.310853 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.310851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a2d78-df53-4a75-9c87-2a69fb360230-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.311093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.310877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.311093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.311004 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.311093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.311021 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.311093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.311043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcw5\" (UniqueName: \"kubernetes.io/projected/ba8a2d78-df53-4a75-9c87-2a69fb360230-kube-api-access-kvcw5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412267 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412221 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412267 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a2d78-df53-4a75-9c87-2a69fb360230-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412525 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412525 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412525 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412525 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412457 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcw5\" (UniqueName: \"kubernetes.io/projected/ba8a2d78-df53-4a75-9c87-2a69fb360230-kube-api-access-kvcw5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412784 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412660 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412831 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412787 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.412872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.412838 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.414664 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.414645 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8a2d78-df53-4a75-9c87-2a69fb360230-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.414947 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.414929 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8a2d78-df53-4a75-9c87-2a69fb360230-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.420822 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.420796 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcw5\" (UniqueName: \"kubernetes.io/projected/ba8a2d78-df53-4a75-9c87-2a69fb360230-kube-api-access-kvcw5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6trfj\" (UID: \"ba8a2d78-df53-4a75-9c87-2a69fb360230\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.546513 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.546479 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:35.682815 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:35.682666 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj"] Jul 15 19:10:35.685111 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:10:35.685084 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8a2d78_df53_4a75_9c87_2a69fb360230.slice/crio-89d6188a870152aa7655d56452e80b217160c8ab9b109f08c97949fb5b782a83 WatchSource:0}: Error finding container 89d6188a870152aa7655d56452e80b217160c8ab9b109f08c97949fb5b782a83: Status 404 returned error can't find the container with id 89d6188a870152aa7655d56452e80b217160c8ab9b109f08c97949fb5b782a83 Jul 15 19:10:36.295288 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:36.295181 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:10:36.663476 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:36.663385 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" event={"ID":"ba8a2d78-df53-4a75-9c87-2a69fb360230","Type":"ContainerStarted","Data":"89d6188a870152aa7655d56452e80b217160c8ab9b109f08c97949fb5b782a83"} Jul 15 19:10:37.669783 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.669747 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" event={"ID":"ba8a2d78-df53-4a75-9c87-2a69fb360230","Type":"ContainerStarted","Data":"2226114e7ef7aaf10c6b14077709879b0a3ac97d5247807d388094910854c449"} Jul 15 19:10:37.692872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.692811 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" podStartSLOduration=1.488913956 podStartE2EDuration="2.692793704s" totalImagesPullingTime="1.203879748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:10:37.691436295 +0000 UTC m=+553.524937847" watchObservedRunningTime="2026-07-15 19:10:37.692793704 +0000 UTC m=+553.526295233" Jul 15 19:10:37.891305 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.891259 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j"] Jul 15 19:10:37.894136 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.894113 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:37.902899 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.902878 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 15 19:10:37.933023 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.932931 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j"] Jul 15 19:10:37.941383 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.941348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjj2z\" (UniqueName: \"kubernetes.io/projected/e5def47a-4618-466f-888e-904c78274047-kube-api-access-xjj2z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:37.941546 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.941436 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:37.941546 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.941497 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5def47a-4618-466f-888e-904c78274047-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:37.941546 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.941533 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:37.941725 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.941570 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:37.941725 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:37.941629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042180 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042132 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042180 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042184 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042398 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042227 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjj2z\" (UniqueName: \"kubernetes.io/projected/e5def47a-4618-466f-888e-904c78274047-kube-api-access-xjj2z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042398 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042271 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042398 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042306 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5def47a-4618-466f-888e-904c78274047-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042398 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042328 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042659 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042634 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042732 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042672 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.042789 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.042727 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.044525 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.044503 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5def47a-4618-466f-888e-904c78274047-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.044821 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.044801 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5def47a-4618-466f-888e-904c78274047-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.053846 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.053819 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjj2z\" (UniqueName: \"kubernetes.io/projected/e5def47a-4618-466f-888e-904c78274047-kube-api-access-xjj2z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j\" (UID: \"e5def47a-4618-466f-888e-904c78274047\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.204379 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.204288 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:38.345524 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.345496 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j"] Jul 15 19:10:38.347748 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:10:38.347718 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5def47a_4618_466f_888e_904c78274047.slice/crio-3f87ecd7b3e9cf90a462fcce07173d9e8ba0470e52cc99065b775e196ba10236 WatchSource:0}: Error finding container 3f87ecd7b3e9cf90a462fcce07173d9e8ba0470e52cc99065b775e196ba10236: Status 404 returned error can't find the container with id 3f87ecd7b3e9cf90a462fcce07173d9e8ba0470e52cc99065b775e196ba10236 Jul 15 19:10:38.677151 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.677113 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" event={"ID":"e5def47a-4618-466f-888e-904c78274047","Type":"ContainerStarted","Data":"8b2c697800e3ae738284f24cc6afd10c7ca5ec07857a097216f867623fbaaea0"} Jul 15 19:10:38.677597 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.677178 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" event={"ID":"e5def47a-4618-466f-888e-904c78274047","Type":"ContainerStarted","Data":"3f87ecd7b3e9cf90a462fcce07173d9e8ba0470e52cc99065b775e196ba10236"} Jul 15 19:10:38.698903 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:38.698846 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" podStartSLOduration=1.500877979 podStartE2EDuration="1.698827857s" totalImagesPullingTime="197.949878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:10:38.696992638 +0000 UTC m=+554.530494191" watchObservedRunningTime="2026-07-15 19:10:38.698827857 +0000 UTC m=+554.532329615" Jul 15 19:10:43.395504 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:43.395470 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:10:45.547343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:45.547293 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:45.547794 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:45.547378 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:45.559810 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:45.559782 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:45.721345 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:45.721308 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6trfj" Jul 15 19:10:48.204544 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:48.204509 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:48.204934 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:48.204564 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:48.216822 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:48.216800 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:48.735888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:48.735860 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j" Jul 15 19:10:51.756161 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.756127 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh"] Jul 15 19:10:51.758475 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.758452 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.761893 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.761868 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 19:10:51.775118 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.775089 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh"] Jul 15 19:10:51.875751 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.875711 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.875751 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.875757 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgk7r\" (UniqueName: \"kubernetes.io/projected/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-kube-api-access-cgk7r\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.876022 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.875852 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.876022 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.875918 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.876022 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.875974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.876166 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.876032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.976543 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.976489 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.976543 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.976542 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgk7r\" (UniqueName: \"kubernetes.io/projected/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-kube-api-access-cgk7r\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.976843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.976592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.976843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.976667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.976843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.976726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.976843 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.976766 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.977049 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.977002 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.977090 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.977045 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.977130 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.977107 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.978953 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.978930 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.979111 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.979093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:51.988747 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:51.988721 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgk7r\" (UniqueName: \"kubernetes.io/projected/a69f1bf4-a8f1-4dac-b930-a0a884854bb8-kube-api-access-cgk7r\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh\" (UID: \"a69f1bf4-a8f1-4dac-b930-a0a884854bb8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:52.068480 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:52.068436 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:10:52.202565 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:52.202539 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh"] Jul 15 19:10:52.205311 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:10:52.205287 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda69f1bf4_a8f1_4dac_b930_a0a884854bb8.slice/crio-48baf58fdb0814b42dfc2a69b6231a2ce0770b7bffe8560f9dca78f4abebd253 WatchSource:0}: Error finding container 48baf58fdb0814b42dfc2a69b6231a2ce0770b7bffe8560f9dca78f4abebd253: Status 404 returned error can't find the container with id 48baf58fdb0814b42dfc2a69b6231a2ce0770b7bffe8560f9dca78f4abebd253 Jul 15 19:10:52.745834 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:52.745739 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" event={"ID":"a69f1bf4-a8f1-4dac-b930-a0a884854bb8","Type":"ContainerStarted","Data":"7742c95eaa2c15f6877c16211216520f0737b414c54df7196df1a033a16bf9af"} Jul 15 19:10:52.745834 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:52.745782 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" event={"ID":"a69f1bf4-a8f1-4dac-b930-a0a884854bb8","Type":"ContainerStarted","Data":"48baf58fdb0814b42dfc2a69b6231a2ce0770b7bffe8560f9dca78f4abebd253"} Jul 15 19:10:52.763097 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:52.763042 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" podStartSLOduration=1.534796726 podStartE2EDuration="1.763026179s" totalImagesPullingTime="228.229453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:10:52.762010228 +0000 UTC m=+568.595511781" watchObservedRunningTime="2026-07-15 19:10:52.763026179 +0000 UTC m=+568.596527731" Jul 15 19:10:53.294430 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:10:53.294394 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:11:02.069300 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:11:02.069242 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:11:02.069300 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:11:02.069320 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:11:02.081996 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:11:02.081969 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:11:02.802987 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:11:02.802953 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh" Jul 15 19:11:24.744575 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:11:24.744535 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:11:24.745589 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:11:24.745570 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:12:36.490754 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:12:36.490716 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:12:47.194085 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:12:47.194045 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:12:58.290724 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:12:58.290683 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:13:07.296356 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:13:07.296320 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:13:18.185121 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:13:18.185088 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:13:44.893135 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:13:44.893093 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:13:58.585101 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:13:58.585065 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:14:31.092120 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:14:31.092077 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:14:38.586044 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:14:38.586007 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:14:52.697428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:14:52.697330 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:15:03.493076 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:15:03.493038 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:15:13.288493 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:15:13.288458 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:15:22.791138 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:15:22.791104 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:15:33.496696 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:15:33.495626 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:15:43.990085 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:15:43.990049 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:16:08.998524 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:16:08.998487 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:16:18.194128 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:16:18.194043 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:16:24.782077 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:16:24.782049 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:16:24.782532 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:16:24.782298 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:16:42.496640 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:16:42.496589 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:16:58.587253 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:16:58.587218 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:17:08.303941 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:17:08.303907 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:17:20.698709 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:17:20.698669 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:17:35.084382 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:17:35.084339 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:19:34.698568 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:19:34.698527 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:19:46.446767 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:19:46.446705 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:19:56.095546 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:19:56.095511 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:20:08.999799 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:20:08.999759 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:20:18.893986 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:20:18.893949 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:20:30.197125 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:20:30.197089 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:20:38.592198 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:20:38.592164 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:20:55.607461 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:20:55.607378 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:21:04.694850 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:21:04.694812 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:21:15.323459 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:21:15.323420 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:21:24.816674 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:21:24.816645 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:21:24.818028 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:21:24.818007 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:21:26.301581 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:21:26.301551 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:21:49.715583 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:21:49.715541 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:22:00.509714 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:22:00.509679 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:22:10.513012 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:22:10.512976 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:22:20.900850 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:22:20.900761 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:22:29.599029 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:22:29.598992 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:22:41.826042 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:22:41.826007 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:22:51.995064 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:22:51.995028 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:23:00.998334 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:23:00.997619 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:23:11.999986 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:23:11.999949 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:23:21.608505 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:23:21.608457 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:23:32.411132 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:23:32.411091 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:23:44.016292 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:23:44.016259 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:23:59.413736 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:23:59.413654 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:24:08.900865 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:24:08.900833 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:24:15.539064 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:24:15.539021 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:24:25.899362 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:24:25.899326 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:24:36.610419 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:24:36.610372 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:24:50.205963 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:24:50.205917 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:25:00.994337 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:25:00.994304 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:25:12.096754 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:25:12.096710 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:25:25.022136 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:25:25.022054 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:25:36.897643 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:25:36.897594 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:25:49.034809 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:25:49.034769 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:26:00.296407 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:26:00.296370 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:26:10.014592 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:26:10.014501 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:26:20.116616 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:26:20.116579 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:26:24.848205 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:26:24.848174 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:26:24.849890 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:26:24.849866 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:28:06.787751 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:28:06.787708 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:28:14.493490 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:28:14.493449 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:28:23.617299 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:28:23.617202 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:28:33.785388 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:28:33.785335 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:28:44.396428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:28:44.396389 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:29:11.988880 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:29:11.988846 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:29:24.189830 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:29:24.189793 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:30:12.405375 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:12.405319 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:30:23.786653 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:23.785658 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:30:30.965888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:30.965847 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz"] Jul 15 19:30:30.969528 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:30.969503 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:30.974001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:30.972214 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4256e5\"/\"default-dockercfg-qdgkn\"" Jul 15 19:30:30.974001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:30.973217 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4256e5\"/\"test-model-4256e5-kserve-self-signed-certs\"" Jul 15 19:30:30.974001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:30.973498 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4256e5\"/\"openshift-service-ca.crt\"" Jul 15 19:30:30.974001 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:30.973775 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4256e5\"/\"kube-root-ca.crt\"" Jul 15 19:30:30.989696 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:30.989655 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz"] Jul 15 19:30:31.035713 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.035665 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-tmp-dir\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.035886 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.035825 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-model-cache\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.035927 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.035889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cea02a3-5157-462c-811b-3c5133aca358-tls-certs\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.035966 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.035933 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-home\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.035999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.035961 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfbs\" (UniqueName: \"kubernetes.io/projected/1cea02a3-5157-462c-811b-3c5133aca358-kube-api-access-8rfbs\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.036033 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.036012 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-dshm\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.137377 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137342 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-tmp-dir\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.137559 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137412 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-model-cache\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.137559 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cea02a3-5157-462c-811b-3c5133aca358-tls-certs\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.137559 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137463 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-home\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.137559 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137482 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfbs\" (UniqueName: \"kubernetes.io/projected/1cea02a3-5157-462c-811b-3c5133aca358-kube-api-access-8rfbs\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.137559 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137514 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-dshm\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.137930 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137902 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-home\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.138006 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.137952 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-model-cache\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.138046 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.138020 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-tmp-dir\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.140360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.140333 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-dshm\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.140645 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.140595 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cea02a3-5157-462c-811b-3c5133aca358-tls-certs\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.145911 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.145886 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfbs\" (UniqueName: \"kubernetes.io/projected/1cea02a3-5157-462c-811b-3c5133aca358-kube-api-access-8rfbs\") pod \"test-model-4256e5-kserve-784c99974c-2kflz\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.288319 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.288278 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:31.426480 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.426448 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz"] Jul 15 19:30:31.429139 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:30:31.429089 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cea02a3_5157_462c_811b_3c5133aca358.slice/crio-79eb29669a6f653abb8f284c5dbe206240acf472027a3f0a8a36125b2beccc96 WatchSource:0}: Error finding container 79eb29669a6f653abb8f284c5dbe206240acf472027a3f0a8a36125b2beccc96: Status 404 returned error can't find the container with id 79eb29669a6f653abb8f284c5dbe206240acf472027a3f0a8a36125b2beccc96 Jul 15 19:30:31.540166 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.540073 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" event={"ID":"1cea02a3-5157-462c-811b-3c5133aca358","Type":"ContainerStarted","Data":"09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3"} Jul 15 19:30:31.540166 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.540117 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" event={"ID":"1cea02a3-5157-462c-811b-3c5133aca358","Type":"ContainerStarted","Data":"79eb29669a6f653abb8f284c5dbe206240acf472027a3f0a8a36125b2beccc96"} Jul 15 19:30:31.563254 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:31.563165 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" podStartSLOduration=1.563142854 podStartE2EDuration="1.563142854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:31.562824078 +0000 UTC m=+1747.396325631" watchObservedRunningTime="2026-07-15 19:30:31.563142854 +0000 UTC m=+1747.396644418" Jul 15 19:30:36.490066 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:36.490032 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:30:41.289439 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:41.289379 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:41.289917 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:41.289529 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:41.302501 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:41.302472 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:30:41.601018 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:30:41.600934 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:31:02.365895 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.365858 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k"] Jul 15 19:31:02.374092 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.374060 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.377965 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.377938 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-381b50\"/\"openshift-service-ca.crt\"" Jul 15 19:31:02.377965 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.377961 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-381b50\"/\"test-model-381b50-kserve-self-signed-certs\"" Jul 15 19:31:02.379052 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.379029 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-381b50\"/\"default-dockercfg-dmch5\"" Jul 15 19:31:02.379181 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.379135 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-381b50\"/\"kube-root-ca.crt\"" Jul 15 19:31:02.382913 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.382874 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k"] Jul 15 19:31:02.446111 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.446072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-home\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.446301 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.446125 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-dshm\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.446301 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.446246 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tmp-dir\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.446301 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.446286 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5km9h\" (UniqueName: \"kubernetes.io/projected/69fb1095-edfd-48a1-8dfb-df770ac1dc76-kube-api-access-5km9h\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.446409 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.446342 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-model-cache\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.446409 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.446357 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tls-certs\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.547872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.547833 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-home\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.547872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.547882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-dshm\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.548189 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.547944 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tmp-dir\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.548189 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.547974 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km9h\" (UniqueName: \"kubernetes.io/projected/69fb1095-edfd-48a1-8dfb-df770ac1dc76-kube-api-access-5km9h\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.548189 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.548016 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-model-cache\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.548189 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.548039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tls-certs\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.548405 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.548295 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-home\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.548405 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.548362 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tmp-dir\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.548478 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.548422 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-model-cache\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.550307 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.550284 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-dshm\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.550504 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.550490 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tls-certs\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.556351 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.556325 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km9h\" (UniqueName: \"kubernetes.io/projected/69fb1095-edfd-48a1-8dfb-df770ac1dc76-kube-api-access-5km9h\") pod \"test-model-381b50-kserve-648cd65979-wdr7k\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.686818 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.686721 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:02.827549 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:02.827522 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k"] Jul 15 19:31:02.830468 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:31:02.830432 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69fb1095_edfd_48a1_8dfb_df770ac1dc76.slice/crio-aab03694b2e50df8ee699360d4807d68d686ca40990359e8b2f68e7cc17da478 WatchSource:0}: Error finding container aab03694b2e50df8ee699360d4807d68d686ca40990359e8b2f68e7cc17da478: Status 404 returned error can't find the container with id aab03694b2e50df8ee699360d4807d68d686ca40990359e8b2f68e7cc17da478 Jul 15 19:31:03.700447 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:03.700410 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" event={"ID":"69fb1095-edfd-48a1-8dfb-df770ac1dc76","Type":"ContainerStarted","Data":"18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe"} Jul 15 19:31:03.700447 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:03.700452 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" event={"ID":"69fb1095-edfd-48a1-8dfb-df770ac1dc76","Type":"ContainerStarted","Data":"aab03694b2e50df8ee699360d4807d68d686ca40990359e8b2f68e7cc17da478"} Jul 15 19:31:03.737316 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:03.737259 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" podStartSLOduration=1.7372434399999999 podStartE2EDuration="1.73724344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:31:03.734861736 +0000 UTC m=+1779.568363307" watchObservedRunningTime="2026-07-15 19:31:03.73724344 +0000 UTC m=+1779.570744989" Jul 15 19:31:06.587573 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:06.587537 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:31:12.687130 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:12.687083 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:12.687702 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:12.687154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:12.699842 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:12.699813 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:12.757567 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:12.757532 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:20.184733 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:20.184650 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:31:24.880299 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:24.880270 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:31:24.882731 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:24.882706 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:31:26.110170 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:26.110133 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz"] Jul 15 19:31:27.023799 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:27.023765 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k"] Jul 15 19:31:33.493035 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:33.492932 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:31:41.130962 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.130914 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" podUID="1cea02a3-5157-462c-811b-3c5133aca358" containerName="main" containerID="cri-o://09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3" gracePeriod=45 Jul 15 19:31:41.401003 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.400975 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:31:41.556087 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556056 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-tmp-dir\") pod \"1cea02a3-5157-462c-811b-3c5133aca358\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " Jul 15 19:31:41.556260 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556133 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-home\") pod \"1cea02a3-5157-462c-811b-3c5133aca358\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " Jul 15 19:31:41.556260 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556178 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-dshm\") pod \"1cea02a3-5157-462c-811b-3c5133aca358\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " Jul 15 19:31:41.556260 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556229 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cea02a3-5157-462c-811b-3c5133aca358-tls-certs\") pod \"1cea02a3-5157-462c-811b-3c5133aca358\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " Jul 15 19:31:41.556377 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556261 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rfbs\" (UniqueName: \"kubernetes.io/projected/1cea02a3-5157-462c-811b-3c5133aca358-kube-api-access-8rfbs\") pod \"1cea02a3-5157-462c-811b-3c5133aca358\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " Jul 15 19:31:41.556377 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556290 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-model-cache\") pod \"1cea02a3-5157-462c-811b-3c5133aca358\" (UID: \"1cea02a3-5157-462c-811b-3c5133aca358\") " Jul 15 19:31:41.556470 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556388 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cea02a3-5157-462c-811b-3c5133aca358" (UID: "1cea02a3-5157-462c-811b-3c5133aca358"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:41.556470 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556436 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-home" (OuterVolumeSpecName: "home") pod "1cea02a3-5157-462c-811b-3c5133aca358" (UID: "1cea02a3-5157-462c-811b-3c5133aca358"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:41.556657 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556634 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cea02a3-5157-462c-811b-3c5133aca358" (UID: "1cea02a3-5157-462c-811b-3c5133aca358"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:41.556734 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556689 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:41.556734 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.556712 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:41.558505 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.558479 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-dshm" (OuterVolumeSpecName: "dshm") pod "1cea02a3-5157-462c-811b-3c5133aca358" (UID: "1cea02a3-5157-462c-811b-3c5133aca358"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:41.558641 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.558522 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cea02a3-5157-462c-811b-3c5133aca358-kube-api-access-8rfbs" (OuterVolumeSpecName: "kube-api-access-8rfbs") pod "1cea02a3-5157-462c-811b-3c5133aca358" (UID: "1cea02a3-5157-462c-811b-3c5133aca358"). InnerVolumeSpecName "kube-api-access-8rfbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:31:41.558641 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.558595 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cea02a3-5157-462c-811b-3c5133aca358-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cea02a3-5157-462c-811b-3c5133aca358" (UID: "1cea02a3-5157-462c-811b-3c5133aca358"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:31:41.657644 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.657508 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:41.657644 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.657545 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cea02a3-5157-462c-811b-3c5133aca358-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:41.657644 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.657560 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rfbs\" (UniqueName: \"kubernetes.io/projected/1cea02a3-5157-462c-811b-3c5133aca358-kube-api-access-8rfbs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:41.657644 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.657572 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cea02a3-5157-462c-811b-3c5133aca358-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:41.897982 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.897947 2561 generic.go:356] "Generic (PLEG): container finished" podID="1cea02a3-5157-462c-811b-3c5133aca358" containerID="09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3" exitCode=0 Jul 15 19:31:41.898168 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.898016 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" Jul 15 19:31:41.898168 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.898031 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" event={"ID":"1cea02a3-5157-462c-811b-3c5133aca358","Type":"ContainerDied","Data":"09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3"} Jul 15 19:31:41.898168 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.898080 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz" event={"ID":"1cea02a3-5157-462c-811b-3c5133aca358","Type":"ContainerDied","Data":"79eb29669a6f653abb8f284c5dbe206240acf472027a3f0a8a36125b2beccc96"} Jul 15 19:31:41.898168 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.898098 2561 scope.go:117] "RemoveContainer" containerID="09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3" Jul 15 19:31:41.907698 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.907646 2561 scope.go:117] "RemoveContainer" containerID="09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3" Jul 15 19:31:41.907961 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:31:41.907943 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3\": container with ID starting with 09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3 not found: ID does not exist" containerID="09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3" Jul 15 19:31:41.908006 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.907971 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3"} err="failed to get container status \"09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3\": rpc error: code = NotFound desc = could not find container \"09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3\": container with ID starting with 09ee7b10b71e9a5bfb75b6918ce8e7dcc87639cb66dd470e03e4c02a2b6a87c3 not found: ID does not exist" Jul 15 19:31:41.921580 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.921545 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz"] Jul 15 19:31:41.928366 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:41.928337 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4256e5/test-model-4256e5-kserve-784c99974c-2kflz"] Jul 15 19:31:42.044360 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.044325 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" podUID="69fb1095-edfd-48a1-8dfb-df770ac1dc76" containerName="main" containerID="cri-o://18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe" gracePeriod=45 Jul 15 19:31:42.314616 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.314576 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:42.468873 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.468779 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-model-cache\") pod \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " Jul 15 19:31:42.468873 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.468861 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5km9h\" (UniqueName: \"kubernetes.io/projected/69fb1095-edfd-48a1-8dfb-df770ac1dc76-kube-api-access-5km9h\") pod \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " Jul 15 19:31:42.469093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.468926 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tmp-dir\") pod \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " Jul 15 19:31:42.469093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.468952 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-home\") pod \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " Jul 15 19:31:42.469093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.468997 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tls-certs\") pod \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " Jul 15 19:31:42.469093 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.469028 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-dshm\") pod \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\" (UID: \"69fb1095-edfd-48a1-8dfb-df770ac1dc76\") " Jul 15 19:31:42.469307 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.469110 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-model-cache" (OuterVolumeSpecName: "model-cache") pod "69fb1095-edfd-48a1-8dfb-df770ac1dc76" (UID: "69fb1095-edfd-48a1-8dfb-df770ac1dc76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:42.469307 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.469233 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69fb1095-edfd-48a1-8dfb-df770ac1dc76" (UID: "69fb1095-edfd-48a1-8dfb-df770ac1dc76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:42.469307 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.469248 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-home" (OuterVolumeSpecName: "home") pod "69fb1095-edfd-48a1-8dfb-df770ac1dc76" (UID: "69fb1095-edfd-48a1-8dfb-df770ac1dc76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:42.469431 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.469355 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:42.469431 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.469374 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:42.469431 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.469386 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:42.471150 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.471123 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-dshm" (OuterVolumeSpecName: "dshm") pod "69fb1095-edfd-48a1-8dfb-df770ac1dc76" (UID: "69fb1095-edfd-48a1-8dfb-df770ac1dc76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:31:42.471248 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.471125 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69fb1095-edfd-48a1-8dfb-df770ac1dc76" (UID: "69fb1095-edfd-48a1-8dfb-df770ac1dc76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:31:42.471248 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.471204 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fb1095-edfd-48a1-8dfb-df770ac1dc76-kube-api-access-5km9h" (OuterVolumeSpecName: "kube-api-access-5km9h") pod "69fb1095-edfd-48a1-8dfb-df770ac1dc76" (UID: "69fb1095-edfd-48a1-8dfb-df770ac1dc76"). InnerVolumeSpecName "kube-api-access-5km9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:31:42.570974 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.570930 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5km9h\" (UniqueName: \"kubernetes.io/projected/69fb1095-edfd-48a1-8dfb-df770ac1dc76-kube-api-access-5km9h\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:42.570974 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.570968 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fb1095-edfd-48a1-8dfb-df770ac1dc76-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:42.570974 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.570977 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fb1095-edfd-48a1-8dfb-df770ac1dc76-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:31:42.905443 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.905408 2561 generic.go:356] "Generic (PLEG): container finished" podID="69fb1095-edfd-48a1-8dfb-df770ac1dc76" containerID="18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe" exitCode=0 Jul 15 19:31:42.905443 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.905444 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" Jul 15 19:31:42.905745 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.905493 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" event={"ID":"69fb1095-edfd-48a1-8dfb-df770ac1dc76","Type":"ContainerDied","Data":"18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe"} Jul 15 19:31:42.905745 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.905533 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k" event={"ID":"69fb1095-edfd-48a1-8dfb-df770ac1dc76","Type":"ContainerDied","Data":"aab03694b2e50df8ee699360d4807d68d686ca40990359e8b2f68e7cc17da478"} Jul 15 19:31:42.905745 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.905556 2561 scope.go:117] "RemoveContainer" containerID="18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe" Jul 15 19:31:42.915513 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.915490 2561 scope.go:117] "RemoveContainer" containerID="18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe" Jul 15 19:31:42.915831 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:31:42.915811 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe\": container with ID starting with 18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe not found: ID does not exist" containerID="18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe" Jul 15 19:31:42.915898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.915844 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe"} err="failed to get container status \"18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe\": rpc error: code = NotFound desc = could not find container \"18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe\": container with ID starting with 18bdddd9bb4effb5cc9564787be6883658d4fd75b1c2dd8474092992e9003cfe not found: ID does not exist" Jul 15 19:31:42.934392 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.934360 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cea02a3-5157-462c-811b-3c5133aca358" path="/var/lib/kubelet/pods/1cea02a3-5157-462c-811b-3c5133aca358/volumes" Jul 15 19:31:42.934673 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.934658 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k"] Jul 15 19:31:42.935307 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:42.935280 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-381b50/test-model-381b50-kserve-648cd65979-wdr7k"] Jul 15 19:31:44.933465 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:31:44.933421 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69fb1095-edfd-48a1-8dfb-df770ac1dc76" path="/var/lib/kubelet/pods/69fb1095-edfd-48a1-8dfb-df770ac1dc76/volumes" Jul 15 19:32:02.590923 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:32:02.590882 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:32:15.890024 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:32:15.889979 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:32:26.991453 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:32:26.991415 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:32:37.728352 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:32:37.728313 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:32:54.784637 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:32:54.784538 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:33:03.692748 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:03.692710 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:33:13.090543 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:13.090496 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:33:27.492531 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:27.492490 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:33:40.263269 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.263234 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g"] Jul 15 19:33:40.263807 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.263643 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cea02a3-5157-462c-811b-3c5133aca358" containerName="main" Jul 15 19:33:40.263807 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.263659 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cea02a3-5157-462c-811b-3c5133aca358" containerName="main" Jul 15 19:33:40.263807 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.263687 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fb1095-edfd-48a1-8dfb-df770ac1dc76" containerName="main" Jul 15 19:33:40.263807 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.263693 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fb1095-edfd-48a1-8dfb-df770ac1dc76" containerName="main" Jul 15 19:33:40.263807 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.263772 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cea02a3-5157-462c-811b-3c5133aca358" containerName="main" Jul 15 19:33:40.263807 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.263781 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fb1095-edfd-48a1-8dfb-df770ac1dc76" containerName="main" Jul 15 19:33:40.266636 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.266602 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.269127 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.269103 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"kube-root-ca.crt\"" Jul 15 19:33:40.269127 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.269116 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"openshift-service-ca.crt\"" Jul 15 19:33:40.270031 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.270012 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"auth-test-model-4a4ed2-kserve-self-signed-certs\"" Jul 15 19:33:40.270112 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.270013 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"default-dockercfg-zqnxm\"" Jul 15 19:33:40.278391 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.278366 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g"] Jul 15 19:33:40.344229 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.344195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-model-cache\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.344410 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.344239 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-home\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.344410 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.344270 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-dshm\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.344410 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.344292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-tmp-dir\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.344410 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.344338 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6fn6\" (UniqueName: \"kubernetes.io/projected/4fff4c6f-4045-4009-95bb-61f86fe38877-kube-api-access-z6fn6\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.344410 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.344390 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fff4c6f-4045-4009-95bb-61f86fe38877-tls-certs\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.445657 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.445586 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-model-cache\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.445849 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.445673 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-home\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.445849 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.445701 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-dshm\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.445849 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.445721 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-tmp-dir\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.445849 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.445781 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6fn6\" (UniqueName: \"kubernetes.io/projected/4fff4c6f-4045-4009-95bb-61f86fe38877-kube-api-access-z6fn6\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.445849 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.445824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fff4c6f-4045-4009-95bb-61f86fe38877-tls-certs\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.446102 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.446059 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-model-cache\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.446102 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.446083 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-home\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.446241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.446218 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-tmp-dir\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.448056 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.448038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-dshm\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.448316 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.448294 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fff4c6f-4045-4009-95bb-61f86fe38877-tls-certs\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.454898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.454874 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6fn6\" (UniqueName: \"kubernetes.io/projected/4fff4c6f-4045-4009-95bb-61f86fe38877-kube-api-access-z6fn6\") pod \"auth-test-model-4a4ed2-kserve-667967f587-74l8g\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.577193 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.577153 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:40.724219 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:40.724193 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g"] Jul 15 19:33:40.727309 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:33:40.727280 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fff4c6f_4045_4009_95bb_61f86fe38877.slice/crio-4c82c3b8a4e9678f9bea8a0a1f8be7857166386f28215de762e418819f5abc27 WatchSource:0}: Error finding container 4c82c3b8a4e9678f9bea8a0a1f8be7857166386f28215de762e418819f5abc27: Status 404 returned error can't find the container with id 4c82c3b8a4e9678f9bea8a0a1f8be7857166386f28215de762e418819f5abc27 Jul 15 19:33:41.494302 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:41.494263 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" event={"ID":"4fff4c6f-4045-4009-95bb-61f86fe38877","Type":"ContainerStarted","Data":"9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224"} Jul 15 19:33:41.494302 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:41.494310 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" event={"ID":"4fff4c6f-4045-4009-95bb-61f86fe38877","Type":"ContainerStarted","Data":"4c82c3b8a4e9678f9bea8a0a1f8be7857166386f28215de762e418819f5abc27"} Jul 15 19:33:41.515285 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:41.515221 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" podStartSLOduration=1.5151981970000001 podStartE2EDuration="1.515198197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:33:41.513387747 +0000 UTC m=+1937.346889301" watchObservedRunningTime="2026-07-15 19:33:41.515198197 +0000 UTC m=+1937.348699750" Jul 15 19:33:43.755234 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:43.755198 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:33:50.577966 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:50.577919 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:50.578470 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:50.577980 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:50.590697 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:50.590667 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:33:51.554070 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:33:51.554041 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:34:09.671996 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.671955 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt"] Jul 15 19:34:09.675681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.675662 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.678948 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.678918 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"openshift-service-ca.crt\"" Jul 15 19:34:09.679074 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.678918 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"default-dockercfg-j95w5\"" Jul 15 19:34:09.679775 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.679755 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"kube-root-ca.crt\"" Jul 15 19:34:09.679939 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.679802 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"auth-test-model-f83827-kserve-self-signed-certs\"" Jul 15 19:34:09.693784 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.693759 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt"] Jul 15 19:34:09.844910 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.844873 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46013086-56e9-465f-b665-01de04044122-tls-certs\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.845095 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.844938 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjn4x\" (UniqueName: \"kubernetes.io/projected/46013086-56e9-465f-b665-01de04044122-kube-api-access-hjn4x\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.845095 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.844971 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-home\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.845095 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.844993 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-tmp-dir\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.845095 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.845056 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-model-cache\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.845250 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.845133 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-dshm\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.945996 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.945897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-dshm\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.945996 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.945993 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46013086-56e9-465f-b665-01de04044122-tls-certs\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.946245 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.946026 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjn4x\" (UniqueName: \"kubernetes.io/projected/46013086-56e9-465f-b665-01de04044122-kube-api-access-hjn4x\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.946245 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.946048 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-home\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.946245 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.946068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-tmp-dir\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.946245 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.946091 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-model-cache\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.946479 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.946436 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-home\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.946523 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.946486 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-tmp-dir\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.946523 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.946510 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-model-cache\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.948241 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.948221 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-dshm\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.948395 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.948378 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46013086-56e9-465f-b665-01de04044122-tls-certs\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.962268 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.962233 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjn4x\" (UniqueName: \"kubernetes.io/projected/46013086-56e9-465f-b665-01de04044122-kube-api-access-hjn4x\") pod \"auth-test-model-f83827-kserve-679fd65db7-qdtrt\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:09.985668 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:09.985636 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:10.143582 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:10.143441 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt"] Jul 15 19:34:10.146244 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:34:10.146220 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46013086_56e9_465f_b665_01de04044122.slice/crio-e51a196fc24bdc3f14e2e94f3f80e304ec05745eb9c524612e2d0f80d8283c39 WatchSource:0}: Error finding container e51a196fc24bdc3f14e2e94f3f80e304ec05745eb9c524612e2d0f80d8283c39: Status 404 returned error can't find the container with id e51a196fc24bdc3f14e2e94f3f80e304ec05745eb9c524612e2d0f80d8283c39 Jul 15 19:34:10.641487 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:10.641453 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" event={"ID":"46013086-56e9-465f-b665-01de04044122","Type":"ContainerStarted","Data":"e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde"} Jul 15 19:34:10.641691 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:10.641504 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" event={"ID":"46013086-56e9-465f-b665-01de04044122","Type":"ContainerStarted","Data":"e51a196fc24bdc3f14e2e94f3f80e304ec05745eb9c524612e2d0f80d8283c39"} Jul 15 19:34:10.664729 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:10.664675 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" podStartSLOduration=1.664654648 podStartE2EDuration="1.664654648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:10.664247676 +0000 UTC m=+1966.497749229" watchObservedRunningTime="2026-07-15 19:34:10.664654648 +0000 UTC m=+1966.498156203" Jul 15 19:34:13.408302 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:13.408260 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:34:19.986343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:19.986238 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:19.986343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:19.986305 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:19.998911 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:19.998883 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:20.703516 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:20.703488 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:34:27.784025 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:27.783991 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:34:42.295454 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:34:42.295416 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:35:01.487549 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:35:01.487515 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:35:20.002564 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:35:20.002526 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:35:38.313959 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:35:38.313910 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:35:52.606278 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:35:52.606192 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:36:12.196158 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:12.196117 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:36:16.796797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:16.796760 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g"] Jul 15 19:36:17.653018 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:17.652982 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt"] Jul 15 19:36:24.912035 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:24.911996 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:36:24.916354 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:24.916330 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kc4vn_7818af85-6523-469d-b917-243de65f815f/ovn-acl-logging/0.log" Jul 15 19:36:27.892331 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:27.892289 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:36:31.817273 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:31.817200 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" podUID="4fff4c6f-4045-4009-95bb-61f86fe38877" containerName="main" containerID="cri-o://9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224" gracePeriod=45 Jul 15 19:36:32.076453 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.076388 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:36:32.194037 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.193996 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6fn6\" (UniqueName: \"kubernetes.io/projected/4fff4c6f-4045-4009-95bb-61f86fe38877-kube-api-access-z6fn6\") pod \"4fff4c6f-4045-4009-95bb-61f86fe38877\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " Jul 15 19:36:32.194243 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194065 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fff4c6f-4045-4009-95bb-61f86fe38877-tls-certs\") pod \"4fff4c6f-4045-4009-95bb-61f86fe38877\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " Jul 15 19:36:32.194243 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194097 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-home\") pod \"4fff4c6f-4045-4009-95bb-61f86fe38877\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " Jul 15 19:36:32.194243 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194169 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-dshm\") pod \"4fff4c6f-4045-4009-95bb-61f86fe38877\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " Jul 15 19:36:32.194243 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194218 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-model-cache\") pod \"4fff4c6f-4045-4009-95bb-61f86fe38877\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " Jul 15 19:36:32.194454 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194241 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-tmp-dir\") pod \"4fff4c6f-4045-4009-95bb-61f86fe38877\" (UID: \"4fff4c6f-4045-4009-95bb-61f86fe38877\") " Jul 15 19:36:32.194454 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194410 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-home" (OuterVolumeSpecName: "home") pod "4fff4c6f-4045-4009-95bb-61f86fe38877" (UID: "4fff4c6f-4045-4009-95bb-61f86fe38877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:32.194569 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194500 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fff4c6f-4045-4009-95bb-61f86fe38877" (UID: "4fff4c6f-4045-4009-95bb-61f86fe38877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:32.194707 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194686 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:32.194787 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194730 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fff4c6f-4045-4009-95bb-61f86fe38877" (UID: "4fff4c6f-4045-4009-95bb-61f86fe38877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:32.194787 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.194739 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:32.196470 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.196446 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fff4c6f-4045-4009-95bb-61f86fe38877-kube-api-access-z6fn6" (OuterVolumeSpecName: "kube-api-access-z6fn6") pod "4fff4c6f-4045-4009-95bb-61f86fe38877" (UID: "4fff4c6f-4045-4009-95bb-61f86fe38877"). InnerVolumeSpecName "kube-api-access-z6fn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:36:32.196600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.196472 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fff4c6f-4045-4009-95bb-61f86fe38877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fff4c6f-4045-4009-95bb-61f86fe38877" (UID: "4fff4c6f-4045-4009-95bb-61f86fe38877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:36:32.196600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.196477 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-dshm" (OuterVolumeSpecName: "dshm") pod "4fff4c6f-4045-4009-95bb-61f86fe38877" (UID: "4fff4c6f-4045-4009-95bb-61f86fe38877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:32.295761 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.295723 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:32.295761 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.295752 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fff4c6f-4045-4009-95bb-61f86fe38877-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:32.295761 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.295765 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6fn6\" (UniqueName: \"kubernetes.io/projected/4fff4c6f-4045-4009-95bb-61f86fe38877-kube-api-access-z6fn6\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:32.295761 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.295773 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fff4c6f-4045-4009-95bb-61f86fe38877-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:32.360717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.360589 2561 generic.go:356] "Generic (PLEG): container finished" podID="4fff4c6f-4045-4009-95bb-61f86fe38877" containerID="9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224" exitCode=0 Jul 15 19:36:32.360717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.360653 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" Jul 15 19:36:32.360717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.360680 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" event={"ID":"4fff4c6f-4045-4009-95bb-61f86fe38877","Type":"ContainerDied","Data":"9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224"} Jul 15 19:36:32.360978 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.360728 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g" event={"ID":"4fff4c6f-4045-4009-95bb-61f86fe38877","Type":"ContainerDied","Data":"4c82c3b8a4e9678f9bea8a0a1f8be7857166386f28215de762e418819f5abc27"} Jul 15 19:36:32.360978 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.360749 2561 scope.go:117] "RemoveContainer" containerID="9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224" Jul 15 19:36:32.372861 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.372836 2561 scope.go:117] "RemoveContainer" containerID="9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224" Jul 15 19:36:32.373136 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:36:32.373119 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224\": container with ID starting with 9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224 not found: ID does not exist" containerID="9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224" Jul 15 19:36:32.373201 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.373144 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224"} err="failed to get container status \"9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224\": rpc error: code = NotFound desc = could not find container \"9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224\": container with ID starting with 9ac234c81d84897f4cdfde4f72fe26b7be2f872caa099541bd58fbb90037f224 not found: ID does not exist" Jul 15 19:36:32.386260 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.386228 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g"] Jul 15 19:36:32.393214 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.393174 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/auth-test-model-4a4ed2-kserve-667967f587-74l8g"] Jul 15 19:36:32.678385 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.678269 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" podUID="46013086-56e9-465f-b665-01de04044122" containerName="main" containerID="cri-o://e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde" gracePeriod=45 Jul 15 19:36:32.927271 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.927237 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:36:32.932670 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:32.932587 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fff4c6f-4045-4009-95bb-61f86fe38877" path="/var/lib/kubelet/pods/4fff4c6f-4045-4009-95bb-61f86fe38877/volumes" Jul 15 19:36:33.003285 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003243 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-dshm\") pod \"46013086-56e9-465f-b665-01de04044122\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " Jul 15 19:36:33.003499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003338 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-home\") pod \"46013086-56e9-465f-b665-01de04044122\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " Jul 15 19:36:33.003499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003388 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-tmp-dir\") pod \"46013086-56e9-465f-b665-01de04044122\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " Jul 15 19:36:33.003499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003441 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-model-cache\") pod \"46013086-56e9-465f-b665-01de04044122\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " Jul 15 19:36:33.003499 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003491 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjn4x\" (UniqueName: \"kubernetes.io/projected/46013086-56e9-465f-b665-01de04044122-kube-api-access-hjn4x\") pod \"46013086-56e9-465f-b665-01de04044122\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " Jul 15 19:36:33.003764 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003516 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46013086-56e9-465f-b665-01de04044122-tls-certs\") pod \"46013086-56e9-465f-b665-01de04044122\" (UID: \"46013086-56e9-465f-b665-01de04044122\") " Jul 15 19:36:33.003764 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003634 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-home" (OuterVolumeSpecName: "home") pod "46013086-56e9-465f-b665-01de04044122" (UID: "46013086-56e9-465f-b665-01de04044122"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:33.003764 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003659 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46013086-56e9-465f-b665-01de04044122" (UID: "46013086-56e9-465f-b665-01de04044122"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:33.003933 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.003787 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-model-cache" (OuterVolumeSpecName: "model-cache") pod "46013086-56e9-465f-b665-01de04044122" (UID: "46013086-56e9-465f-b665-01de04044122"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:33.004039 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.004007 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:33.004102 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.004049 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:33.004102 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.004066 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:33.006073 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.006040 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46013086-56e9-465f-b665-01de04044122-kube-api-access-hjn4x" (OuterVolumeSpecName: "kube-api-access-hjn4x") pod "46013086-56e9-465f-b665-01de04044122" (UID: "46013086-56e9-465f-b665-01de04044122"). InnerVolumeSpecName "kube-api-access-hjn4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:36:33.006210 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.006122 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46013086-56e9-465f-b665-01de04044122-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46013086-56e9-465f-b665-01de04044122" (UID: "46013086-56e9-465f-b665-01de04044122"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:36:33.006210 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.006135 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-dshm" (OuterVolumeSpecName: "dshm") pod "46013086-56e9-465f-b665-01de04044122" (UID: "46013086-56e9-465f-b665-01de04044122"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:36:33.105485 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.105452 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjn4x\" (UniqueName: \"kubernetes.io/projected/46013086-56e9-465f-b665-01de04044122-kube-api-access-hjn4x\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:33.105485 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.105479 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46013086-56e9-465f-b665-01de04044122-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:33.105485 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.105490 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46013086-56e9-465f-b665-01de04044122-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:36:33.369512 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.369479 2561 generic.go:356] "Generic (PLEG): container finished" podID="46013086-56e9-465f-b665-01de04044122" containerID="e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde" exitCode=0 Jul 15 19:36:33.369716 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.369522 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" Jul 15 19:36:33.369716 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.369540 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" event={"ID":"46013086-56e9-465f-b665-01de04044122","Type":"ContainerDied","Data":"e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde"} Jul 15 19:36:33.369716 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.369571 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt" event={"ID":"46013086-56e9-465f-b665-01de04044122","Type":"ContainerDied","Data":"e51a196fc24bdc3f14e2e94f3f80e304ec05745eb9c524612e2d0f80d8283c39"} Jul 15 19:36:33.369716 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.369587 2561 scope.go:117] "RemoveContainer" containerID="e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde" Jul 15 19:36:33.379005 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.378982 2561 scope.go:117] "RemoveContainer" containerID="e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde" Jul 15 19:36:33.379371 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:36:33.379349 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde\": container with ID starting with e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde not found: ID does not exist" containerID="e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde" Jul 15 19:36:33.379450 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.379387 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde"} err="failed to get container status \"e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde\": rpc error: code = NotFound desc = could not find container \"e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde\": container with ID starting with e92c0cb5c6988874b12e54e83b1319e5d08fd0395d402155a2fb1e1d32560cde not found: ID does not exist" Jul 15 19:36:33.392459 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.392428 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt"] Jul 15 19:36:33.397116 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:33.397088 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/auth-test-model-f83827-kserve-679fd65db7-qdtrt"] Jul 15 19:36:34.934154 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:34.934122 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46013086-56e9-465f-b665-01de04044122" path="/var/lib/kubelet/pods/46013086-56e9-465f-b665-01de04044122/volumes" Jul 15 19:36:48.743250 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.743218 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8"] Jul 15 19:36:48.743763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.743646 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46013086-56e9-465f-b665-01de04044122" containerName="main" Jul 15 19:36:48.743763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.743663 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="46013086-56e9-465f-b665-01de04044122" containerName="main" Jul 15 19:36:48.743763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.743674 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fff4c6f-4045-4009-95bb-61f86fe38877" containerName="main" Jul 15 19:36:48.743763 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.743680 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fff4c6f-4045-4009-95bb-61f86fe38877" containerName="main" Jul 15 19:36:48.744002 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.743782 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="46013086-56e9-465f-b665-01de04044122" containerName="main" Jul 15 19:36:48.744002 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.743796 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fff4c6f-4045-4009-95bb-61f86fe38877" containerName="main" Jul 15 19:36:48.747949 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.747929 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.750487 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.750463 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"openshift-service-ca.crt\"" Jul 15 19:36:48.751428 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.751403 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"kube-root-ca.crt\"" Jul 15 19:36:48.751533 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.751408 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"sub-test-model-f83827-kserve-self-signed-certs\"" Jul 15 19:36:48.751533 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.751443 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f83827\"/\"default-dockercfg-j95w5\"" Jul 15 19:36:48.758565 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.758540 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8"] Jul 15 19:36:48.869701 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.869664 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqlz\" (UniqueName: \"kubernetes.io/projected/d6569003-72ea-47b0-a6e1-38cbb89652b4-kube-api-access-9qqlz\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.869877 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.869726 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-tmp-dir\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.869877 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.869810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6569003-72ea-47b0-a6e1-38cbb89652b4-tls-certs\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.869877 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.869851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-model-cache\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.869978 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.869893 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-home\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.869978 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.869945 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-dshm\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.970656 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.970584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqlz\" (UniqueName: \"kubernetes.io/projected/d6569003-72ea-47b0-a6e1-38cbb89652b4-kube-api-access-9qqlz\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.970859 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.970673 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-tmp-dir\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.970859 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.970700 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6569003-72ea-47b0-a6e1-38cbb89652b4-tls-certs\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.970859 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.970722 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-model-cache\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.970859 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.970742 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-home\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.970859 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.970791 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-dshm\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.971137 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.971105 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-tmp-dir\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.971198 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.971156 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-home\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.971261 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.971234 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-model-cache\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.973074 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.973052 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-dshm\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.973315 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.973299 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6569003-72ea-47b0-a6e1-38cbb89652b4-tls-certs\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:48.979230 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:48.979209 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqlz\" (UniqueName: \"kubernetes.io/projected/d6569003-72ea-47b0-a6e1-38cbb89652b4-kube-api-access-9qqlz\") pod \"sub-test-model-f83827-kserve-84654dc995-dz6c8\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:49.058366 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:49.058329 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:49.202010 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:49.201943 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8"] Jul 15 19:36:49.205062 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:36:49.205036 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6569003_72ea_47b0_a6e1_38cbb89652b4.slice/crio-d60b0dcb2caab100866b7d57549a9350ec026b92fe3f9a27d9c828843b26e1ef WatchSource:0}: Error finding container d60b0dcb2caab100866b7d57549a9350ec026b92fe3f9a27d9c828843b26e1ef: Status 404 returned error can't find the container with id d60b0dcb2caab100866b7d57549a9350ec026b92fe3f9a27d9c828843b26e1ef Jul 15 19:36:49.461207 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:49.461106 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" event={"ID":"d6569003-72ea-47b0-a6e1-38cbb89652b4","Type":"ContainerStarted","Data":"767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f"} Jul 15 19:36:49.461207 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:49.461149 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" event={"ID":"d6569003-72ea-47b0-a6e1-38cbb89652b4","Type":"ContainerStarted","Data":"d60b0dcb2caab100866b7d57549a9350ec026b92fe3f9a27d9c828843b26e1ef"} Jul 15 19:36:49.481667 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:49.481594 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" podStartSLOduration=1.481578525 podStartE2EDuration="1.481578525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:36:49.479782969 +0000 UTC m=+2125.313284538" watchObservedRunningTime="2026-07-15 19:36:49.481578525 +0000 UTC m=+2125.315080077" Jul 15 19:36:53.196232 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:53.196196 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:36:59.059410 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:59.059362 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:59.059922 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:59.059638 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:59.072530 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:59.072503 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:36:59.522032 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:36:59.522006 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:37:15.194931 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:15.194896 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:37:31.401140 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:31.401102 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:37:38.359752 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:38.359718 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8"] Jul 15 19:37:49.292886 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:49.292844 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:37:52.354389 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.354354 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565"] Jul 15 19:37:52.358041 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.358023 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.363518 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.363486 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"default-dockercfg-zqnxm\"" Jul 15 19:37:52.363518 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.363491 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"openshift-service-ca.crt\"" Jul 15 19:37:52.363754 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.363566 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"rate-test-model-4a4ed2-kserve-self-signed-certs\"" Jul 15 19:37:52.363754 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.363620 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4a4ed2\"/\"kube-root-ca.crt\"" Jul 15 19:37:52.372049 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.372022 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565"] Jul 15 19:37:52.500254 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.500211 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-tmp-dir\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.500254 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.500258 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-dshm\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.500483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.500289 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt72v\" (UniqueName: \"kubernetes.io/projected/b9b290c3-4578-4589-bc11-fc07d761d2f5-kube-api-access-pt72v\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.500483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.500324 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b290c3-4578-4589-bc11-fc07d761d2f5-tls-certs\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.500483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.500399 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-home\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.500483 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.500432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-model-cache\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.601936 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.601892 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-tmp-dir\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.601936 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.601936 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-dshm\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.601936 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.601957 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt72v\" (UniqueName: \"kubernetes.io/projected/b9b290c3-4578-4589-bc11-fc07d761d2f5-kube-api-access-pt72v\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.602314 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.601993 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b290c3-4578-4589-bc11-fc07d761d2f5-tls-certs\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.602314 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.602036 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-home\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.602314 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.602072 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-model-cache\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.602492 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.602469 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-home\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.602557 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.602495 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-model-cache\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.602630 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.602586 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-tmp-dir\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.604594 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.604534 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-dshm\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.604731 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.604682 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b290c3-4578-4589-bc11-fc07d761d2f5-tls-certs\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.613290 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.613263 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt72v\" (UniqueName: \"kubernetes.io/projected/b9b290c3-4578-4589-bc11-fc07d761d2f5-kube-api-access-pt72v\") pod \"rate-test-model-4a4ed2-kserve-cfd6d7787-67565\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.667440 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.667400 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:37:52.809315 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:52.809268 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565"] Jul 15 19:37:52.812344 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:37:52.812311 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b290c3_4578_4589_bc11_fc07d761d2f5.slice/crio-ec3cb4349b21c74ea96ac8e08341ece0ba7c0a6313cf9374c787560d7a149ac5 WatchSource:0}: Error finding container ec3cb4349b21c74ea96ac8e08341ece0ba7c0a6313cf9374c787560d7a149ac5: Status 404 returned error can't find the container with id ec3cb4349b21c74ea96ac8e08341ece0ba7c0a6313cf9374c787560d7a149ac5 Jul 15 19:37:53.382226 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.382154 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" podUID="d6569003-72ea-47b0-a6e1-38cbb89652b4" containerName="main" containerID="cri-o://767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f" gracePeriod=45 Jul 15 19:37:53.645944 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.645877 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:37:53.716491 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716462 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6569003-72ea-47b0-a6e1-38cbb89652b4-tls-certs\") pod \"d6569003-72ea-47b0-a6e1-38cbb89652b4\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " Jul 15 19:37:53.716717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716516 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-model-cache\") pod \"d6569003-72ea-47b0-a6e1-38cbb89652b4\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " Jul 15 19:37:53.716717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716536 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-dshm\") pod \"d6569003-72ea-47b0-a6e1-38cbb89652b4\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " Jul 15 19:37:53.716717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716567 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qqlz\" (UniqueName: \"kubernetes.io/projected/d6569003-72ea-47b0-a6e1-38cbb89652b4-kube-api-access-9qqlz\") pod \"d6569003-72ea-47b0-a6e1-38cbb89652b4\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " Jul 15 19:37:53.716717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716660 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-tmp-dir\") pod \"d6569003-72ea-47b0-a6e1-38cbb89652b4\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " Jul 15 19:37:53.716717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716676 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-home\") pod \"d6569003-72ea-47b0-a6e1-38cbb89652b4\" (UID: \"d6569003-72ea-47b0-a6e1-38cbb89652b4\") " Jul 15 19:37:53.717017 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716814 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6569003-72ea-47b0-a6e1-38cbb89652b4" (UID: "d6569003-72ea-47b0-a6e1-38cbb89652b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:53.717017 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.716944 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6569003-72ea-47b0-a6e1-38cbb89652b4" (UID: "d6569003-72ea-47b0-a6e1-38cbb89652b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:53.717129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.717009 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-home" (OuterVolumeSpecName: "home") pod "d6569003-72ea-47b0-a6e1-38cbb89652b4" (UID: "d6569003-72ea-47b0-a6e1-38cbb89652b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:53.717129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.717092 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:37:53.717129 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.717111 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:37:53.718797 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.718768 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6569003-72ea-47b0-a6e1-38cbb89652b4-kube-api-access-9qqlz" (OuterVolumeSpecName: "kube-api-access-9qqlz") pod "d6569003-72ea-47b0-a6e1-38cbb89652b4" (UID: "d6569003-72ea-47b0-a6e1-38cbb89652b4"). InnerVolumeSpecName "kube-api-access-9qqlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:37:53.718951 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.718807 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6569003-72ea-47b0-a6e1-38cbb89652b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6569003-72ea-47b0-a6e1-38cbb89652b4" (UID: "d6569003-72ea-47b0-a6e1-38cbb89652b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:37:53.718951 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.718811 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-dshm" (OuterVolumeSpecName: "dshm") pod "d6569003-72ea-47b0-a6e1-38cbb89652b4" (UID: "d6569003-72ea-47b0-a6e1-38cbb89652b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:53.788403 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.788359 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" event={"ID":"b9b290c3-4578-4589-bc11-fc07d761d2f5","Type":"ContainerStarted","Data":"bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5"} Jul 15 19:37:53.788403 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.788409 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" event={"ID":"b9b290c3-4578-4589-bc11-fc07d761d2f5","Type":"ContainerStarted","Data":"ec3cb4349b21c74ea96ac8e08341ece0ba7c0a6313cf9374c787560d7a149ac5"} Jul 15 19:37:53.789780 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.789752 2561 generic.go:356] "Generic (PLEG): container finished" podID="d6569003-72ea-47b0-a6e1-38cbb89652b4" containerID="767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f" exitCode=0 Jul 15 19:37:53.789898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.789785 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" Jul 15 19:37:53.789898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.789835 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" event={"ID":"d6569003-72ea-47b0-a6e1-38cbb89652b4","Type":"ContainerDied","Data":"767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f"} Jul 15 19:37:53.789898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.789871 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8" event={"ID":"d6569003-72ea-47b0-a6e1-38cbb89652b4","Type":"ContainerDied","Data":"d60b0dcb2caab100866b7d57549a9350ec026b92fe3f9a27d9c828843b26e1ef"} Jul 15 19:37:53.789898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.789888 2561 scope.go:117] "RemoveContainer" containerID="767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f" Jul 15 19:37:53.799895 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.799871 2561 scope.go:117] "RemoveContainer" containerID="767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f" Jul 15 19:37:53.800183 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:37:53.800167 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f\": container with ID starting with 767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f not found: ID does not exist" containerID="767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f" Jul 15 19:37:53.800236 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.800191 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f"} err="failed to get container status \"767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f\": rpc error: code = NotFound desc = could not find container \"767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f\": container with ID starting with 767ea4d92d2b2aa2b84d65d31d3fd3544d45cbb98857a098c7376b780ca8674f not found: ID does not exist" Jul 15 19:37:53.806188 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.806137 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" podStartSLOduration=1.806118901 podStartE2EDuration="1.806118901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:53.804919648 +0000 UTC m=+2189.638421203" watchObservedRunningTime="2026-07-15 19:37:53.806118901 +0000 UTC m=+2189.639620472" Jul 15 19:37:53.817888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.817810 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:37:53.817888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.817841 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6569003-72ea-47b0-a6e1-38cbb89652b4-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:37:53.817888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.817854 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6569003-72ea-47b0-a6e1-38cbb89652b4-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:37:53.817888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.817866 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qqlz\" (UniqueName: \"kubernetes.io/projected/d6569003-72ea-47b0-a6e1-38cbb89652b4-kube-api-access-9qqlz\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:37:53.819600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.819567 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8"] Jul 15 19:37:53.821863 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:53.821839 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f83827/sub-test-model-f83827-kserve-84654dc995-dz6c8"] Jul 15 19:37:54.933440 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:37:54.933405 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6569003-72ea-47b0-a6e1-38cbb89652b4" path="/var/lib/kubelet/pods/d6569003-72ea-47b0-a6e1-38cbb89652b4/volumes" Jul 15 19:38:02.667830 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:38:02.667768 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:38:02.667830 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:38:02.667846 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:38:02.680578 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:38:02.680549 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:38:02.848174 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:38:02.848136 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:38:05.196869 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:38:05.196832 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:38:22.797424 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:38:22.797387 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:38:51.206504 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:38:51.206419 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:39:06.188515 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:06.188474 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:39:12.897717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:12.897681 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565"] Jul 15 19:39:14.739372 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.739334 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139723952439472"] Jul 15 19:39:14.739796 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.739774 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6569003-72ea-47b0-a6e1-38cbb89652b4" containerName="main" Jul 15 19:39:14.739796 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.739787 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6569003-72ea-47b0-a6e1-38cbb89652b4" containerName="main" Jul 15 19:39:14.739876 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.739858 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6569003-72ea-47b0-a6e1-38cbb89652b4" containerName="main" Jul 15 19:39:14.743184 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.743163 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:14.746098 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.746076 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bd68r\"" Jul 15 19:39:14.752664 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.752636 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723952439472"] Jul 15 19:39:14.844966 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.844925 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8r8m\" (UniqueName: \"kubernetes.io/projected/13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e-kube-api-access-n8r8m\") pod \"test-curl-7742-139723952439472\" (UID: \"13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e\") " pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:14.946382 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.946346 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8r8m\" (UniqueName: \"kubernetes.io/projected/13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e-kube-api-access-n8r8m\") pod \"test-curl-7742-139723952439472\" (UID: \"13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e\") " pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:14.957011 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:14.956973 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8r8m\" (UniqueName: \"kubernetes.io/projected/13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e-kube-api-access-n8r8m\") pod \"test-curl-7742-139723952439472\" (UID: \"13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e\") " pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:15.053330 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:15.053274 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:15.195179 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:15.195156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723952439472"] Jul 15 19:39:15.197742 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:39:15.197714 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13d27bc4_4aaa_4b04_b5fd_bea2a3d3176e.slice/crio-ac2403f616a638d966932a25dfcd79dcafb7a3efc40855b6fd6cc6a8428f4ccb WatchSource:0}: Error finding container ac2403f616a638d966932a25dfcd79dcafb7a3efc40855b6fd6cc6a8428f4ccb: Status 404 returned error can't find the container with id ac2403f616a638d966932a25dfcd79dcafb7a3efc40855b6fd6cc6a8428f4ccb Jul 15 19:39:15.199123 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:15.199103 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:39:15.209981 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:15.209952 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723952439472" event={"ID":"13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e","Type":"ContainerStarted","Data":"ac2403f616a638d966932a25dfcd79dcafb7a3efc40855b6fd6cc6a8428f4ccb"} Jul 15 19:39:16.215825 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:16.215729 2561 generic.go:356] "Generic (PLEG): container finished" podID="13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e" containerID="843f5f74b5eb98397815532e89f5d8756c6c4bb0e05d2ef17d725367f22523e7" exitCode=0 Jul 15 19:39:16.215825 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:16.215813 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723952439472" event={"ID":"13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e","Type":"ContainerDied","Data":"843f5f74b5eb98397815532e89f5d8756c6c4bb0e05d2ef17d725367f22523e7"} Jul 15 19:39:17.359213 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.359187 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:17.366137 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.366112 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8r8m\" (UniqueName: \"kubernetes.io/projected/13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e-kube-api-access-n8r8m\") pod \"13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e\" (UID: \"13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e\") " Jul 15 19:39:17.368366 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.368340 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e-kube-api-access-n8r8m" (OuterVolumeSpecName: "kube-api-access-n8r8m") pod "13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e" (UID: "13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e"). InnerVolumeSpecName "kube-api-access-n8r8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:17.396867 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.396842 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139723952439472_13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e/test-curl-7742-139723952439472/0.log" Jul 15 19:39:17.420929 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.420891 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139723952439472"] Jul 15 19:39:17.424653 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.424622 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139723952439472"] Jul 15 19:39:17.467372 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.467326 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8r8m\" (UniqueName: \"kubernetes.io/projected/13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e-kube-api-access-n8r8m\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:17.548493 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.548455 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139723952388400"] Jul 15 19:39:17.549898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.549860 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e" containerName="test-curl-7742-139723952439472" Jul 15 19:39:17.549898 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.549898 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e" containerName="test-curl-7742-139723952439472" Jul 15 19:39:17.550088 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.550068 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e" containerName="test-curl-7742-139723952439472" Jul 15 19:39:17.557498 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.557369 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952388400" Jul 15 19:39:17.559736 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.559707 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723952388400"] Jul 15 19:39:17.567862 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.567835 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvh2p\" (UniqueName: \"kubernetes.io/projected/1a232e95-ae01-4075-a9dc-f942aa9c8bc0-kube-api-access-nvh2p\") pod \"test-curl-7742-139723952388400\" (UID: \"1a232e95-ae01-4075-a9dc-f942aa9c8bc0\") " pod="opendatahub/test-curl-7742-139723952388400" Jul 15 19:39:17.668726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.668686 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvh2p\" (UniqueName: \"kubernetes.io/projected/1a232e95-ae01-4075-a9dc-f942aa9c8bc0-kube-api-access-nvh2p\") pod \"test-curl-7742-139723952388400\" (UID: \"1a232e95-ae01-4075-a9dc-f942aa9c8bc0\") " pod="opendatahub/test-curl-7742-139723952388400" Jul 15 19:39:17.678515 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.678474 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvh2p\" (UniqueName: \"kubernetes.io/projected/1a232e95-ae01-4075-a9dc-f942aa9c8bc0-kube-api-access-nvh2p\") pod \"test-curl-7742-139723952388400\" (UID: \"1a232e95-ae01-4075-a9dc-f942aa9c8bc0\") " pod="opendatahub/test-curl-7742-139723952388400" Jul 15 19:39:17.868222 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:17.868125 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952388400" Jul 15 19:39:18.003594 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:18.003568 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723952388400"] Jul 15 19:39:18.005852 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:39:18.005822 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a232e95_ae01_4075_a9dc_f942aa9c8bc0.slice/crio-d059a983a54b53b779b2c10bb2b1b991640e33ec015f1c125c2093cf976ffdc7 WatchSource:0}: Error finding container d059a983a54b53b779b2c10bb2b1b991640e33ec015f1c125c2093cf976ffdc7: Status 404 returned error can't find the container with id d059a983a54b53b779b2c10bb2b1b991640e33ec015f1c125c2093cf976ffdc7 Jul 15 19:39:18.227502 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:18.227399 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723952388400" event={"ID":"1a232e95-ae01-4075-a9dc-f942aa9c8bc0","Type":"ContainerStarted","Data":"d059a983a54b53b779b2c10bb2b1b991640e33ec015f1c125c2093cf976ffdc7"} Jul 15 19:39:18.228677 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:18.228660 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac2403f616a638d966932a25dfcd79dcafb7a3efc40855b6fd6cc6a8428f4ccb" Jul 15 19:39:18.228786 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:18.228691 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:18.239495 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:39:18.239462 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7742-139723952439472\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e" pod="opendatahub/test-curl-7742-139723952439472" Jul 15 19:39:18.932395 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:18.932360 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e" path="/var/lib/kubelet/pods/13d27bc4-4aaa-4b04-b5fd-bea2a3d3176e/volumes" Jul 15 19:39:19.235092 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:19.234990 2561 generic.go:356] "Generic (PLEG): container finished" podID="1a232e95-ae01-4075-a9dc-f942aa9c8bc0" containerID="3b5753e3a06dc60b7c66f1673dee78c3458cf8bb416693eeb7a5fbc070dedab5" exitCode=0 Jul 15 19:39:19.235240 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:19.235083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723952388400" event={"ID":"1a232e95-ae01-4075-a9dc-f942aa9c8bc0","Type":"ContainerDied","Data":"3b5753e3a06dc60b7c66f1673dee78c3458cf8bb416693eeb7a5fbc070dedab5"} Jul 15 19:39:20.379111 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.379084 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952388400" Jul 15 19:39:20.397279 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.397252 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvh2p\" (UniqueName: \"kubernetes.io/projected/1a232e95-ae01-4075-a9dc-f942aa9c8bc0-kube-api-access-nvh2p\") pod \"1a232e95-ae01-4075-a9dc-f942aa9c8bc0\" (UID: \"1a232e95-ae01-4075-a9dc-f942aa9c8bc0\") " Jul 15 19:39:20.399400 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.399371 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a232e95-ae01-4075-a9dc-f942aa9c8bc0-kube-api-access-nvh2p" (OuterVolumeSpecName: "kube-api-access-nvh2p") pod "1a232e95-ae01-4075-a9dc-f942aa9c8bc0" (UID: "1a232e95-ae01-4075-a9dc-f942aa9c8bc0"). InnerVolumeSpecName "kube-api-access-nvh2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:20.401298 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.401276 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139723952388400_1a232e95-ae01-4075-a9dc-f942aa9c8bc0/test-curl-7742-139723952388400/0.log" Jul 15 19:39:20.426182 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.426145 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139723952388400"] Jul 15 19:39:20.432368 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.432338 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139723952388400"] Jul 15 19:39:20.498660 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.498551 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvh2p\" (UniqueName: \"kubernetes.io/projected/1a232e95-ae01-4075-a9dc-f942aa9c8bc0-kube-api-access-nvh2p\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:20.932424 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:20.932390 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a232e95-ae01-4075-a9dc-f942aa9c8bc0" path="/var/lib/kubelet/pods/1a232e95-ae01-4075-a9dc-f942aa9c8bc0/volumes" Jul 15 19:39:21.246722 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:21.246639 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723952388400" Jul 15 19:39:21.246722 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:21.246668 2561 scope.go:117] "RemoveContainer" containerID="3b5753e3a06dc60b7c66f1673dee78c3458cf8bb416693eeb7a5fbc070dedab5" Jul 15 19:39:21.404944 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:21.404905 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:39:23.364523 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.364487 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:23.364943 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.364902 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a232e95-ae01-4075-a9dc-f942aa9c8bc0" containerName="test-curl-7742-139723952388400" Jul 15 19:39:23.364943 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.364915 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a232e95-ae01-4075-a9dc-f942aa9c8bc0" containerName="test-curl-7742-139723952388400" Jul 15 19:39:23.365016 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.364986 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a232e95-ae01-4075-a9dc-f942aa9c8bc0" containerName="test-curl-7742-139723952388400" Jul 15 19:39:23.369710 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.369691 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:23.373446 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.373420 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bd68r\"" Jul 15 19:39:23.376580 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.376552 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:23.428747 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.428708 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwnl\" (UniqueName: \"kubernetes.io/projected/c4ed3d60-886c-46c8-8c7f-cb3c493024b1-kube-api-access-cvwnl\") pod \"test-curl-7742-139723951193376\" (UID: \"c4ed3d60-886c-46c8-8c7f-cb3c493024b1\") " pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:23.530264 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.530220 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwnl\" (UniqueName: \"kubernetes.io/projected/c4ed3d60-886c-46c8-8c7f-cb3c493024b1-kube-api-access-cvwnl\") pod \"test-curl-7742-139723951193376\" (UID: \"c4ed3d60-886c-46c8-8c7f-cb3c493024b1\") " pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:23.541055 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.541015 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwnl\" (UniqueName: \"kubernetes.io/projected/c4ed3d60-886c-46c8-8c7f-cb3c493024b1-kube-api-access-cvwnl\") pod \"test-curl-7742-139723951193376\" (UID: \"c4ed3d60-886c-46c8-8c7f-cb3c493024b1\") " pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:23.680237 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.680136 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:23.826938 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:23.826902 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:23.830138 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:39:23.830109 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4ed3d60_886c_46c8_8c7f_cb3c493024b1.slice/crio-1d1d081aaaa4f24bf12972db6cf1d8d9eb0df4ad0637cc3bf54a30dccecefebe WatchSource:0}: Error finding container 1d1d081aaaa4f24bf12972db6cf1d8d9eb0df4ad0637cc3bf54a30dccecefebe: Status 404 returned error can't find the container with id 1d1d081aaaa4f24bf12972db6cf1d8d9eb0df4ad0637cc3bf54a30dccecefebe Jul 15 19:39:24.270693 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:24.270657 2561 generic.go:356] "Generic (PLEG): container finished" podID="c4ed3d60-886c-46c8-8c7f-cb3c493024b1" containerID="5c6c4957641483dfa9ca328d305fadcc6130e017fbc8f8cea4da8a5814a00409" exitCode=0 Jul 15 19:39:24.270902 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:24.270745 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723951193376" event={"ID":"c4ed3d60-886c-46c8-8c7f-cb3c493024b1","Type":"ContainerDied","Data":"5c6c4957641483dfa9ca328d305fadcc6130e017fbc8f8cea4da8a5814a00409"} Jul 15 19:39:24.270902 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:24.270783 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723951193376" event={"ID":"c4ed3d60-886c-46c8-8c7f-cb3c493024b1","Type":"ContainerStarted","Data":"1d1d081aaaa4f24bf12972db6cf1d8d9eb0df4ad0637cc3bf54a30dccecefebe"} Jul 15 19:39:25.420302 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.420275 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:25.449580 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.449547 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvwnl\" (UniqueName: \"kubernetes.io/projected/c4ed3d60-886c-46c8-8c7f-cb3c493024b1-kube-api-access-cvwnl\") pod \"c4ed3d60-886c-46c8-8c7f-cb3c493024b1\" (UID: \"c4ed3d60-886c-46c8-8c7f-cb3c493024b1\") " Jul 15 19:39:25.451847 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.451817 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ed3d60-886c-46c8-8c7f-cb3c493024b1-kube-api-access-cvwnl" (OuterVolumeSpecName: "kube-api-access-cvwnl") pod "c4ed3d60-886c-46c8-8c7f-cb3c493024b1" (UID: "c4ed3d60-886c-46c8-8c7f-cb3c493024b1"). InnerVolumeSpecName "kube-api-access-cvwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:25.464125 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.464076 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139723951193376_c4ed3d60-886c-46c8-8c7f-cb3c493024b1/test-curl-7742-139723951193376/0.log" Jul 15 19:39:25.498709 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.498675 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:25.505782 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.505756 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139723951193376"] Jul 15 19:39:25.550988 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.550907 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvwnl\" (UniqueName: \"kubernetes.io/projected/c4ed3d60-886c-46c8-8c7f-cb3c493024b1-kube-api-access-cvwnl\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:25.628221 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.628184 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:25.628638 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.628623 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ed3d60-886c-46c8-8c7f-cb3c493024b1" containerName="test-curl-7742-139723951193376" Jul 15 19:39:25.628638 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.628639 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ed3d60-886c-46c8-8c7f-cb3c493024b1" containerName="test-curl-7742-139723951193376" Jul 15 19:39:25.628730 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.628719 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ed3d60-886c-46c8-8c7f-cb3c493024b1" containerName="test-curl-7742-139723951193376" Jul 15 19:39:25.631946 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.631930 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:25.643149 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.643121 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:25.752390 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.752355 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khct\" (UniqueName: \"kubernetes.io/projected/de1ced5b-addb-4134-b2c1-2a99ce9c1671-kube-api-access-9khct\") pod \"test-curl-7742-139723955159376\" (UID: \"de1ced5b-addb-4134-b2c1-2a99ce9c1671\") " pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:25.853281 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.853200 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khct\" (UniqueName: \"kubernetes.io/projected/de1ced5b-addb-4134-b2c1-2a99ce9c1671-kube-api-access-9khct\") pod \"test-curl-7742-139723955159376\" (UID: \"de1ced5b-addb-4134-b2c1-2a99ce9c1671\") " pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:25.862684 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.862652 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khct\" (UniqueName: \"kubernetes.io/projected/de1ced5b-addb-4134-b2c1-2a99ce9c1671-kube-api-access-9khct\") pod \"test-curl-7742-139723955159376\" (UID: \"de1ced5b-addb-4134-b2c1-2a99ce9c1671\") " pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:25.941623 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:25.941580 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:26.077021 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:26.076987 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:26.079529 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:39:26.079492 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde1ced5b_addb_4134_b2c1_2a99ce9c1671.slice/crio-aae244769b6b41aa5e3a8c73e9af93a0b50a45decc5681585e1d1cc3b2b2123d WatchSource:0}: Error finding container aae244769b6b41aa5e3a8c73e9af93a0b50a45decc5681585e1d1cc3b2b2123d: Status 404 returned error can't find the container with id aae244769b6b41aa5e3a8c73e9af93a0b50a45decc5681585e1d1cc3b2b2123d Jul 15 19:39:26.282680 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:26.282650 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d1d081aaaa4f24bf12972db6cf1d8d9eb0df4ad0637cc3bf54a30dccecefebe" Jul 15 19:39:26.282680 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:26.282661 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:26.283940 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:26.283911 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723955159376" event={"ID":"de1ced5b-addb-4134-b2c1-2a99ce9c1671","Type":"ContainerStarted","Data":"aae244769b6b41aa5e3a8c73e9af93a0b50a45decc5681585e1d1cc3b2b2123d"} Jul 15 19:39:26.293334 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:39:26.293298 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7742-139723951193376\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="c4ed3d60-886c-46c8-8c7f-cb3c493024b1" pod="opendatahub/test-curl-7742-139723951193376" Jul 15 19:39:26.933445 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:26.933408 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ed3d60-886c-46c8-8c7f-cb3c493024b1" path="/var/lib/kubelet/pods/c4ed3d60-886c-46c8-8c7f-cb3c493024b1/volumes" Jul 15 19:39:27.290726 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:27.290696 2561 generic.go:356] "Generic (PLEG): container finished" podID="de1ced5b-addb-4134-b2c1-2a99ce9c1671" containerID="b03bd8864b4733f98d9eb5c49471aafd1cfea7286d2f65508bcf245d93f92697" exitCode=0 Jul 15 19:39:27.290881 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:27.290752 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723955159376" event={"ID":"de1ced5b-addb-4134-b2c1-2a99ce9c1671","Type":"ContainerDied","Data":"b03bd8864b4733f98d9eb5c49471aafd1cfea7286d2f65508bcf245d93f92697"} Jul 15 19:39:27.918400 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:27.918358 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" podUID="b9b290c3-4578-4589-bc11-fc07d761d2f5" containerName="main" containerID="cri-o://bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5" gracePeriod=45 Jul 15 19:39:28.178976 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.178908 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:39:28.279319 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279278 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-model-cache\") pod \"b9b290c3-4578-4589-bc11-fc07d761d2f5\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " Jul 15 19:39:28.279319 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279321 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt72v\" (UniqueName: \"kubernetes.io/projected/b9b290c3-4578-4589-bc11-fc07d761d2f5-kube-api-access-pt72v\") pod \"b9b290c3-4578-4589-bc11-fc07d761d2f5\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " Jul 15 19:39:28.279581 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279368 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-tmp-dir\") pod \"b9b290c3-4578-4589-bc11-fc07d761d2f5\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " Jul 15 19:39:28.279581 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279393 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-home\") pod \"b9b290c3-4578-4589-bc11-fc07d761d2f5\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " Jul 15 19:39:28.279581 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279420 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b290c3-4578-4589-bc11-fc07d761d2f5-tls-certs\") pod \"b9b290c3-4578-4589-bc11-fc07d761d2f5\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " Jul 15 19:39:28.279581 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279458 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-dshm\") pod \"b9b290c3-4578-4589-bc11-fc07d761d2f5\" (UID: \"b9b290c3-4578-4589-bc11-fc07d761d2f5\") " Jul 15 19:39:28.279866 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279704 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9b290c3-4578-4589-bc11-fc07d761d2f5" (UID: "b9b290c3-4578-4589-bc11-fc07d761d2f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:28.279866 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279743 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9b290c3-4578-4589-bc11-fc07d761d2f5" (UID: "b9b290c3-4578-4589-bc11-fc07d761d2f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:28.279866 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279768 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-home" (OuterVolumeSpecName: "home") pod "b9b290c3-4578-4589-bc11-fc07d761d2f5" (UID: "b9b290c3-4578-4589-bc11-fc07d761d2f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:28.279981 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279928 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-model-cache\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:28.279981 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279946 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-tmp-dir\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:28.279981 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.279958 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-home\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:28.281583 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.281561 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-dshm" (OuterVolumeSpecName: "dshm") pod "b9b290c3-4578-4589-bc11-fc07d761d2f5" (UID: "b9b290c3-4578-4589-bc11-fc07d761d2f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:39:28.281669 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.281638 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b290c3-4578-4589-bc11-fc07d761d2f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9b290c3-4578-4589-bc11-fc07d761d2f5" (UID: "b9b290c3-4578-4589-bc11-fc07d761d2f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:39:28.281735 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.281718 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b290c3-4578-4589-bc11-fc07d761d2f5-kube-api-access-pt72v" (OuterVolumeSpecName: "kube-api-access-pt72v") pod "b9b290c3-4578-4589-bc11-fc07d761d2f5" (UID: "b9b290c3-4578-4589-bc11-fc07d761d2f5"). InnerVolumeSpecName "kube-api-access-pt72v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:28.300681 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.300645 2561 generic.go:356] "Generic (PLEG): container finished" podID="b9b290c3-4578-4589-bc11-fc07d761d2f5" containerID="bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5" exitCode=0 Jul 15 19:39:28.300846 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.300692 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" Jul 15 19:39:28.300846 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.300727 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" event={"ID":"b9b290c3-4578-4589-bc11-fc07d761d2f5","Type":"ContainerDied","Data":"bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5"} Jul 15 19:39:28.300846 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.300773 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565" event={"ID":"b9b290c3-4578-4589-bc11-fc07d761d2f5","Type":"ContainerDied","Data":"ec3cb4349b21c74ea96ac8e08341ece0ba7c0a6313cf9374c787560d7a149ac5"} Jul 15 19:39:28.300846 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.300790 2561 scope.go:117] "RemoveContainer" containerID="bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5" Jul 15 19:39:28.310717 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.310695 2561 scope.go:117] "RemoveContainer" containerID="bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5" Jul 15 19:39:28.311069 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:39:28.311046 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5\": container with ID starting with bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5 not found: ID does not exist" containerID="bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5" Jul 15 19:39:28.311152 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.311084 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5"} err="failed to get container status \"bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5\": rpc error: code = NotFound desc = could not find container \"bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5\": container with ID starting with bb0b090204ffa297e695ea63c7554db3f4b21ff5531d855c8fe8e35f3113eea5 not found: ID does not exist" Jul 15 19:39:28.323317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.323274 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565"] Jul 15 19:39:28.327844 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.327818 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4a4ed2/rate-test-model-4a4ed2-kserve-cfd6d7787-67565"] Jul 15 19:39:28.380702 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.380674 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt72v\" (UniqueName: \"kubernetes.io/projected/b9b290c3-4578-4589-bc11-fc07d761d2f5-kube-api-access-pt72v\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:28.380702 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.380702 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b290c3-4578-4589-bc11-fc07d761d2f5-tls-certs\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:28.380914 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.380715 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b290c3-4578-4589-bc11-fc07d761d2f5-dshm\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:28.432143 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.432077 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:28.453749 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.453715 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139723955159376_de1ced5b-addb-4134-b2c1-2a99ce9c1671/test-curl-7742-139723955159376/0.log" Jul 15 19:39:28.476282 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.476247 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:28.479802 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.479773 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:28.481100 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.481083 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9khct\" (UniqueName: \"kubernetes.io/projected/de1ced5b-addb-4134-b2c1-2a99ce9c1671-kube-api-access-9khct\") pod \"de1ced5b-addb-4134-b2c1-2a99ce9c1671\" (UID: \"de1ced5b-addb-4134-b2c1-2a99ce9c1671\") " Jul 15 19:39:28.483084 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.483055 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1ced5b-addb-4134-b2c1-2a99ce9c1671-kube-api-access-9khct" (OuterVolumeSpecName: "kube-api-access-9khct") pod "de1ced5b-addb-4134-b2c1-2a99ce9c1671" (UID: "de1ced5b-addb-4134-b2c1-2a99ce9c1671"). InnerVolumeSpecName "kube-api-access-9khct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:28.582805 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.582766 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9khct\" (UniqueName: \"kubernetes.io/projected/de1ced5b-addb-4134-b2c1-2a99ce9c1671-kube-api-access-9khct\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:28.592840 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.592805 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:28.593196 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.593181 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1ced5b-addb-4134-b2c1-2a99ce9c1671" containerName="test-curl-7742-139723955159376" Jul 15 19:39:28.593244 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.593196 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1ced5b-addb-4134-b2c1-2a99ce9c1671" containerName="test-curl-7742-139723955159376" Jul 15 19:39:28.593244 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.593232 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b290c3-4578-4589-bc11-fc07d761d2f5" containerName="main" Jul 15 19:39:28.593244 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.593238 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b290c3-4578-4589-bc11-fc07d761d2f5" containerName="main" Jul 15 19:39:28.593329 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.593309 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b290c3-4578-4589-bc11-fc07d761d2f5" containerName="main" Jul 15 19:39:28.593329 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.593316 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1ced5b-addb-4134-b2c1-2a99ce9c1671" containerName="test-curl-7742-139723955159376" Jul 15 19:39:28.596597 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.596580 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:28.603496 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.603469 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:28.683541 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.683452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pfd\" (UniqueName: \"kubernetes.io/projected/fcc0c706-60af-46b6-a7ad-e8df0a11bc3c-kube-api-access-59pfd\") pod \"test-curl-7742-139723955159376\" (UID: \"fcc0c706-60af-46b6-a7ad-e8df0a11bc3c\") " pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:28.785056 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.785014 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pfd\" (UniqueName: \"kubernetes.io/projected/fcc0c706-60af-46b6-a7ad-e8df0a11bc3c-kube-api-access-59pfd\") pod \"test-curl-7742-139723955159376\" (UID: \"fcc0c706-60af-46b6-a7ad-e8df0a11bc3c\") " pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:28.794408 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.794372 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pfd\" (UniqueName: \"kubernetes.io/projected/fcc0c706-60af-46b6-a7ad-e8df0a11bc3c-kube-api-access-59pfd\") pod \"test-curl-7742-139723955159376\" (UID: \"fcc0c706-60af-46b6-a7ad-e8df0a11bc3c\") " pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:28.906459 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.906427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:28.933059 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.933024 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b290c3-4578-4589-bc11-fc07d761d2f5" path="/var/lib/kubelet/pods/b9b290c3-4578-4589-bc11-fc07d761d2f5/volumes" Jul 15 19:39:28.933473 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:28.933453 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1ced5b-addb-4134-b2c1-2a99ce9c1671" path="/var/lib/kubelet/pods/de1ced5b-addb-4134-b2c1-2a99ce9c1671/volumes" Jul 15 19:39:29.046942 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:29.046907 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:29.049512 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:39:29.049485 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcc0c706_60af_46b6_a7ad_e8df0a11bc3c.slice/crio-c44cf11d372bea0bec14e26b7632f9757cd5dfebe2639a9a13b99a8423bc42b5 WatchSource:0}: Error finding container c44cf11d372bea0bec14e26b7632f9757cd5dfebe2639a9a13b99a8423bc42b5: Status 404 returned error can't find the container with id c44cf11d372bea0bec14e26b7632f9757cd5dfebe2639a9a13b99a8423bc42b5 Jul 15 19:39:29.307297 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:29.307263 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:29.307754 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:29.307268 2561 scope.go:117] "RemoveContainer" containerID="b03bd8864b4733f98d9eb5c49471aafd1cfea7286d2f65508bcf245d93f92697" Jul 15 19:39:29.308506 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:29.308479 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723955159376" event={"ID":"fcc0c706-60af-46b6-a7ad-e8df0a11bc3c","Type":"ContainerStarted","Data":"c44cf11d372bea0bec14e26b7632f9757cd5dfebe2639a9a13b99a8423bc42b5"} Jul 15 19:39:30.317630 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:30.317581 2561 generic.go:356] "Generic (PLEG): container finished" podID="fcc0c706-60af-46b6-a7ad-e8df0a11bc3c" containerID="fccee323817caaf5d8dc08d346de5df9d1518c71247df6bb95ba364d46851d88" exitCode=0 Jul 15 19:39:30.318010 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:30.317628 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139723955159376" event={"ID":"fcc0c706-60af-46b6-a7ad-e8df0a11bc3c","Type":"ContainerDied","Data":"fccee323817caaf5d8dc08d346de5df9d1518c71247df6bb95ba364d46851d88"} Jul 15 19:39:31.455893 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:31.455864 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:31.477048 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:31.477010 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139723955159376_fcc0c706-60af-46b6-a7ad-e8df0a11bc3c/test-curl-7742-139723955159376/0.log" Jul 15 19:39:31.500874 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:31.500839 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:31.504420 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:31.504387 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139723955159376"] Jul 15 19:39:31.510635 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:31.510592 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59pfd\" (UniqueName: \"kubernetes.io/projected/fcc0c706-60af-46b6-a7ad-e8df0a11bc3c-kube-api-access-59pfd\") pod \"fcc0c706-60af-46b6-a7ad-e8df0a11bc3c\" (UID: \"fcc0c706-60af-46b6-a7ad-e8df0a11bc3c\") " Jul 15 19:39:31.512760 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:31.512733 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc0c706-60af-46b6-a7ad-e8df0a11bc3c-kube-api-access-59pfd" (OuterVolumeSpecName: "kube-api-access-59pfd") pod "fcc0c706-60af-46b6-a7ad-e8df0a11bc3c" (UID: "fcc0c706-60af-46b6-a7ad-e8df0a11bc3c"). InnerVolumeSpecName "kube-api-access-59pfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:39:31.612425 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:31.612334 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59pfd\" (UniqueName: \"kubernetes.io/projected/fcc0c706-60af-46b6-a7ad-e8df0a11bc3c-kube-api-access-59pfd\") on node \"ip-10-0-130-245.ec2.internal\" DevicePath \"\"" Jul 15 19:39:32.328961 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:32.328933 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44cf11d372bea0bec14e26b7632f9757cd5dfebe2639a9a13b99a8423bc42b5" Jul 15 19:39:32.328961 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:32.328941 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:32.338885 ip-10-0-130-245 kubenswrapper[2561]: E0715 19:39:32.338855 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7742-139723955159376\" is forbidden: User \"system:node:ip-10-0-130-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-245.ec2.internal' and this object" podUID="fcc0c706-60af-46b6-a7ad-e8df0a11bc3c" pod="opendatahub/test-curl-7742-139723955159376" Jul 15 19:39:32.932818 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:32.932784 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc0c706-60af-46b6-a7ad-e8df0a11bc3c" path="/var/lib/kubelet/pods/fcc0c706-60af-46b6-a7ad-e8df0a11bc3c/volumes" Jul 15 19:39:42.101376 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:42.101326 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:39:51.892905 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:51.892866 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zgtjg"] Jul 15 19:39:58.358796 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:58.358764 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-v65kz_4a310d8f-211f-4c02-9b96-bb7704cd08f8/copy-manifests/0.log" Jul 15 19:39:58.365709 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:39:58.365675 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-v65kz_4a310d8f-211f-4c02-9b96-bb7704cd08f8/manager/0.log" Jul 15 19:40:00.678919 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:00.678889 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-sspgx_35e8b23c-5d3b-4e48-ad97-6ee7b4c68821/kuadrant-console-plugin/0.log" Jul 15 19:40:01.037549 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:01.037521 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-zgtjg_a41079bb-6cfe-4886-a771-9bf0aff8600d/limitador/0.log" Jul 15 19:40:01.761341 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:01.761308 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-8ljzt_b270c2b6-c25f-47af-a5a7-64a38a31b22c/kube-auth-proxy/0.log" Jul 15 19:40:02.924367 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:02.924338 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6trfj_ba8a2d78-df53-4a75-9c87-2a69fb360230/main/0.log" Jul 15 19:40:03.038872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:03.038844 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtbz2j_e5def47a-4618-466f-888e-904c78274047/main/0.log" Jul 15 19:40:03.280516 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:03.280481 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-frgxh_a69f1bf4-a8f1-4dac-b930-a0a884854bb8/main/0.log" Jul 15 19:40:11.917841 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:11.917809 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fxjc5_707f404d-e473-4ea4-99e1-3bfb155dfd23/global-pull-secret-syncer/0.log" Jul 15 19:40:12.039367 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:12.039334 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g2qkd_7838d9e7-4153-4796-ad6c-532b49b0d483/konnectivity-agent/0.log" Jul 15 19:40:12.060447 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:12.060421 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-245.ec2.internal_a0d945a25ac134216e2142b82a7b9dc1/haproxy/0.log" Jul 15 19:40:17.011239 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:17.011155 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-sspgx_35e8b23c-5d3b-4e48-ad97-6ee7b4c68821/kuadrant-console-plugin/0.log" Jul 15 19:40:17.230142 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:17.230112 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-zgtjg_a41079bb-6cfe-4886-a771-9bf0aff8600d/limitador/0.log" Jul 15 19:40:19.025269 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.025237 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cee02323-f6b2-4f61-91e2-ae47058a9ffe/alertmanager/0.log" Jul 15 19:40:19.052641 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.052599 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cee02323-f6b2-4f61-91e2-ae47058a9ffe/config-reloader/0.log" Jul 15 19:40:19.082224 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.082195 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cee02323-f6b2-4f61-91e2-ae47058a9ffe/kube-rbac-proxy-web/0.log" Jul 15 19:40:19.106792 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.106717 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cee02323-f6b2-4f61-91e2-ae47058a9ffe/kube-rbac-proxy/0.log" Jul 15 19:40:19.133056 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.133023 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cee02323-f6b2-4f61-91e2-ae47058a9ffe/kube-rbac-proxy-metric/0.log" Jul 15 19:40:19.158123 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.158096 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cee02323-f6b2-4f61-91e2-ae47058a9ffe/prom-label-proxy/0.log" Jul 15 19:40:19.190759 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.190731 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cee02323-f6b2-4f61-91e2-ae47058a9ffe/init-config-reloader/0.log" Jul 15 19:40:19.263354 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.263328 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-7r4q5_e6aeae9d-e720-471a-aac8-8d3510c8bba2/kube-state-metrics/0.log" Jul 15 19:40:19.286600 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.286567 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-7r4q5_e6aeae9d-e720-471a-aac8-8d3510c8bba2/kube-rbac-proxy-main/0.log" Jul 15 19:40:19.310299 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.310273 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-7r4q5_e6aeae9d-e720-471a-aac8-8d3510c8bba2/kube-rbac-proxy-self/0.log" Jul 15 19:40:19.352049 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.352020 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8876b4966-2zksb_dca41ec9-d8f4-4dee-a90a-1ab6ab50d415/metrics-server/0.log" Jul 15 19:40:19.390877 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.390807 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-q2h4p_9eebf6ae-0192-4b49-99ca-eba058cd5d4d/monitoring-plugin/0.log" Jul 15 19:40:19.595526 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.595498 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wx6vq_a3276f22-686f-4e8f-b0e2-b88abce74d34/node-exporter/0.log" Jul 15 19:40:19.620380 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.620360 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wx6vq_a3276f22-686f-4e8f-b0e2-b88abce74d34/kube-rbac-proxy/0.log" Jul 15 19:40:19.646251 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.646171 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wx6vq_a3276f22-686f-4e8f-b0e2-b88abce74d34/init-textfile/0.log" Jul 15 19:40:19.677317 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.677286 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tgqtb_b7ddf0cb-ae1e-4743-b46d-40fddf5f5247/kube-rbac-proxy-main/0.log" Jul 15 19:40:19.701842 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.701812 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tgqtb_b7ddf0cb-ae1e-4743-b46d-40fddf5f5247/kube-rbac-proxy-self/0.log" Jul 15 19:40:19.727515 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.727490 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tgqtb_b7ddf0cb-ae1e-4743-b46d-40fddf5f5247/openshift-state-metrics/0.log" Jul 15 19:40:19.773152 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.773121 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_173fec37-9c71-4522-bfac-f5b321c27fb6/prometheus/0.log" Jul 15 19:40:19.805917 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.805887 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_173fec37-9c71-4522-bfac-f5b321c27fb6/config-reloader/0.log" Jul 15 19:40:19.834602 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.834572 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_173fec37-9c71-4522-bfac-f5b321c27fb6/thanos-sidecar/0.log" Jul 15 19:40:19.860490 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.860463 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_173fec37-9c71-4522-bfac-f5b321c27fb6/kube-rbac-proxy-web/0.log" Jul 15 19:40:19.886716 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.886691 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_173fec37-9c71-4522-bfac-f5b321c27fb6/kube-rbac-proxy/0.log" Jul 15 19:40:19.918533 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.918456 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_173fec37-9c71-4522-bfac-f5b321c27fb6/kube-rbac-proxy-thanos/0.log" Jul 15 19:40:19.946969 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.946944 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_173fec37-9c71-4522-bfac-f5b321c27fb6/init-config-reloader/0.log" Jul 15 19:40:19.980343 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:19.980311 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-9j4hf_4fc58bcb-4be3-46a3-a34a-6288a2e53bf2/prometheus-operator/0.log" Jul 15 19:40:20.002872 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.002847 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-9j4hf_4fc58bcb-4be3-46a3-a34a-6288a2e53bf2/kube-rbac-proxy/0.log" Jul 15 19:40:20.059421 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.059393 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df798c7c-n4dtz_885483b6-feb3-4643-9240-86e8d807fb5f/telemeter-client/0.log" Jul 15 19:40:20.083729 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.083705 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df798c7c-n4dtz_885483b6-feb3-4643-9240-86e8d807fb5f/reload/0.log" Jul 15 19:40:20.107024 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.107000 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df798c7c-n4dtz_885483b6-feb3-4643-9240-86e8d807fb5f/kube-rbac-proxy/0.log" Jul 15 19:40:20.139245 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.139220 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d578769-c9q9r_587a96aa-0182-415a-b9b7-26600e6bd76d/thanos-query/0.log" Jul 15 19:40:20.162236 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.162205 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d578769-c9q9r_587a96aa-0182-415a-b9b7-26600e6bd76d/kube-rbac-proxy-web/0.log" Jul 15 19:40:20.187888 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.187814 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d578769-c9q9r_587a96aa-0182-415a-b9b7-26600e6bd76d/kube-rbac-proxy/0.log" Jul 15 19:40:20.210560 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.210528 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d578769-c9q9r_587a96aa-0182-415a-b9b7-26600e6bd76d/prom-label-proxy/0.log" Jul 15 19:40:20.233370 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.233345 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d578769-c9q9r_587a96aa-0182-415a-b9b7-26600e6bd76d/kube-rbac-proxy-rules/0.log" Jul 15 19:40:20.258448 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:20.258417 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d578769-c9q9r_587a96aa-0182-415a-b9b7-26600e6bd76d/kube-rbac-proxy-metrics/0.log" Jul 15 19:40:21.472468 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.472439 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx"] Jul 15 19:40:21.472897 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.472851 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0c706-60af-46b6-a7ad-e8df0a11bc3c" containerName="test-curl-7742-139723955159376" Jul 15 19:40:21.472897 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.472885 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0c706-60af-46b6-a7ad-e8df0a11bc3c" containerName="test-curl-7742-139723955159376" Jul 15 19:40:21.472980 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.472967 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc0c706-60af-46b6-a7ad-e8df0a11bc3c" containerName="test-curl-7742-139723955159376" Jul 15 19:40:21.475923 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.475902 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.478422 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.478405 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-csbwp\"/\"default-dockercfg-cx64j\"" Jul 15 19:40:21.478529 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.478406 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-csbwp\"/\"kube-root-ca.crt\"" Jul 15 19:40:21.478529 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.478406 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-csbwp\"/\"openshift-service-ca.crt\"" Jul 15 19:40:21.485122 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.485097 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx"] Jul 15 19:40:21.610062 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.610028 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-proc\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.610234 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.610096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp24d\" (UniqueName: \"kubernetes.io/projected/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-kube-api-access-hp24d\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.610234 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.610145 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-sys\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.610234 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.610214 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-podres\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.610335 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.610259 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-lib-modules\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711769 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711721 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-podres\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711769 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711777 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-lib-modules\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711811 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-proc\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711900 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-proc\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711900 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-podres\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp24d\" (UniqueName: \"kubernetes.io/projected/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-kube-api-access-hp24d\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711956 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-lib-modules\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-sys\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.711999 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.711991 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-sys\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.721386 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.721358 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp24d\" (UniqueName: \"kubernetes.io/projected/5d9cfabc-657a-4959-afe9-d09dbb1a2e07-kube-api-access-hp24d\") pod \"perf-node-gather-daemonset-j77hx\" (UID: \"5d9cfabc-657a-4959-afe9-d09dbb1a2e07\") " pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.787075 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.787044 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:21.926675 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:21.926638 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx"] Jul 15 19:40:21.930107 ip-10-0-130-245 kubenswrapper[2561]: W0715 19:40:21.930081 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d9cfabc_657a_4959_afe9_d09dbb1a2e07.slice/crio-78dcc17c6847070edeb9c73bd782a0a38712541201b656e165efd24a99a72cac WatchSource:0}: Error finding container 78dcc17c6847070edeb9c73bd782a0a38712541201b656e165efd24a99a72cac: Status 404 returned error can't find the container with id 78dcc17c6847070edeb9c73bd782a0a38712541201b656e165efd24a99a72cac Jul 15 19:40:22.583528 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:22.583490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" event={"ID":"5d9cfabc-657a-4959-afe9-d09dbb1a2e07","Type":"ContainerStarted","Data":"167e5bf95825feb1ad4a36676033540fd1c3bb8fa5fad4e676fa951b1b4f5896"} Jul 15 19:40:22.583995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:22.583583 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" event={"ID":"5d9cfabc-657a-4959-afe9-d09dbb1a2e07","Type":"ContainerStarted","Data":"78dcc17c6847070edeb9c73bd782a0a38712541201b656e165efd24a99a72cac"} Jul 15 19:40:22.583995 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:22.583632 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:22.602952 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:22.602892 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" podStartSLOduration=1.602870944 podStartE2EDuration="1.602870944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:40:22.602186649 +0000 UTC m=+2338.435688201" watchObservedRunningTime="2026-07-15 19:40:22.602870944 +0000 UTC m=+2338.436372497" Jul 15 19:40:22.694923 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:22.694895 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-654f6498c5-x7scl_795a48e6-d05c-4d12-bb36-60c04ab702b6/console/0.log" Jul 15 19:40:24.390631 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:24.390588 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fdhd_9a08c025-91a2-4c90-9f14-9bc9683083d8/dns/0.log" Jul 15 19:40:24.417536 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:24.417509 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fdhd_9a08c025-91a2-4c90-9f14-9bc9683083d8/kube-rbac-proxy/0.log" Jul 15 19:40:24.582719 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:24.582692 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fsdk5_e21744bb-e296-49be-84f0-d6708c0f1459/dns-node-resolver/0.log" Jul 15 19:40:25.265236 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:25.265153 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r67v4_7f9c4b60-829c-47dc-ac71-3798775f5d55/node-ca/0.log" Jul 15 19:40:26.365917 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:26.365885 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-8ljzt_b270c2b6-c25f-47af-a5a7-64a38a31b22c/kube-auth-proxy/0.log" Jul 15 19:40:27.329981 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:27.329954 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nvscg_0bce3637-bc3e-46ff-8456-030ce270bc5b/serve-healthcheck-canary/0.log" Jul 15 19:40:27.861400 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:27.861368 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vtpv_59771a4c-e433-42eb-82c4-86e0af23ba57/kube-rbac-proxy/0.log" Jul 15 19:40:27.886396 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:27.886373 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vtpv_59771a4c-e433-42eb-82c4-86e0af23ba57/exporter/0.log" Jul 15 19:40:27.911918 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:27.911891 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vtpv_59771a4c-e433-42eb-82c4-86e0af23ba57/extractor/0.log" Jul 15 19:40:28.597980 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:28.597954 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-csbwp/perf-node-gather-daemonset-j77hx" Jul 15 19:40:31.610731 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:31.610699 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-v65kz_4a310d8f-211f-4c02-9b96-bb7704cd08f8/manager/0.log" Jul 15 19:40:31.639318 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:31.639292 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c89bff45-v65kz_4a310d8f-211f-4c02-9b96-bb7704cd08f8/copy-manifests/0.log" Jul 15 19:40:32.737645 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:32.737572 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-8ljzt_b270c2b6-c25f-47af-a5a7-64a38a31b22c/kube-auth-proxy/0.log" Jul 15 19:40:32.967820 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:32.967745 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8659b4c98-nphq4_a40ca77b-ed75-4221-a864-0cf4ae47d591/manager/0.log" Jul 15 19:40:34.277935 ip-10-0-130-245 kubenswrapper[2561]: I0715 19:40:34.277898 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f9c67c7d-8ljzt_b270c2b6-c25f-47af-a5a7-64a38a31b22c/kube-auth-proxy/0.log"