Jul 30 16:58:24.912473 ip-10-0-130-254 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 30 16:58:24.912486 ip-10-0-130-254 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 30 16:58:24.912496 ip-10-0-130-254 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 30 16:58:24.912803 ip-10-0-130-254 systemd[1]: Failed to start Kubernetes Kubelet. Jul 30 16:58:35.152744 ip-10-0-130-254 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 30 16:58:35.152760 ip-10-0-130-254 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2587b801b1ea41e798329927132a3033 -- Jul 30 17:00:30.211633 ip-10-0-130-254 systemd[1]: Starting Kubernetes Kubelet... Jul 30 17:00:30.642447 ip-10-0-130-254 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 17:00:30.642447 ip-10-0-130-254 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 17:00:30.642447 ip-10-0-130-254 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 17:00:30.642447 ip-10-0-130-254 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 17:00:30.642447 ip-10-0-130-254 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 17:00:30.644706 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.644623 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 17:00:30.648837 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648822 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 17:00:30.648837 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648836 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648840 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648844 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648848 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648852 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648857 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648860 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648863 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648866 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648873 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648876 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648879 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648881 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648887 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648890 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648893 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648895 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648898 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648900 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 17:00:30.648903 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648903 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648906 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648909 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648912 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648915 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648918 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648920 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648923 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648927 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648931 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648934 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648936 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648939 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648941 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648943 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648946 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648949 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648951 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648954 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648956 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 17:00:30.649409 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648958 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648961 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648987 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648991 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648994 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648997 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.648999 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649002 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649004 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649006 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649009 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649011 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649014 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649016 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649019 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649022 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649025 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649029 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649031 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 17:00:30.649912 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649034 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649037 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649040 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649042 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649045 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649047 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649049 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649052 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649054 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649057 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649059 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649062 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649064 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649067 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649069 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649072 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649075 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649077 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649080 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649082 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 17:00:30.650370 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649085 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649087 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649090 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649092 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649095 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649097 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649100 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649104 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649107 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649109 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649112 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649114 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649116 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649119 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649121 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649124 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649126 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649129 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649131 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649134 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 17:00:30.651103 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649823 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649829 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649832 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649835 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649837 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649840 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649842 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649845 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649848 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649852 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649855 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649858 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649861 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649863 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649866 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649869 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649871 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649874 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649876 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649880 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 17:00:30.651850 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649883 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649885 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649888 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649890 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649893 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649895 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649898 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649901 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649905 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649909 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649912 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649915 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649918 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649921 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649925 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649929 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649932 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649934 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649937 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 17:00:30.652403 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649939 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649942 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649945 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649948 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649951 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649953 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649956 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649959 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649961 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649965 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649969 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649972 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649976 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649979 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649981 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649984 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649986 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649989 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649992 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 17:00:30.652933 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649994 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649996 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.649999 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650001 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650004 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650007 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650009 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650011 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650014 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650016 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650019 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650021 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650024 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650026 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650028 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650031 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650033 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650036 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650039 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 17:00:30.653452 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650041 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650044 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650046 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650048 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650051 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650054 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650056 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650059 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650061 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650064 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650067 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650070 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650074 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650076 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650079 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650081 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650084 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650086 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650089 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650091 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 17:00:30.653922 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650094 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.650097 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652066 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652079 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652084 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652090 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652094 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652098 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652102 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652106 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652111 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652114 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652118 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652123 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652127 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652131 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652135 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652140 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652144 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652148 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 17:00:30.654414 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652152 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652156 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652161 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652165 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652169 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652173 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652177 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652184 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652190 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652194 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652203 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652207 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652212 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652216 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652220 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652224 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652227 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652231 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652237 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652241 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 17:00:30.654899 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652246 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652249 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652252 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652255 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652257 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652260 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652262 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652265 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652268 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652270 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652273 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652277 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652280 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652284 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652286 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652289 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652291 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652294 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652296 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 17:00:30.655363 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652299 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652301 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652304 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652306 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652308 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652311 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652313 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652316 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652318 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652320 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652324 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652327 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652329 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652334 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652338 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652341 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652343 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652347 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652349 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 17:00:30.655835 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652352 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652355 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652357 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652360 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652363 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652366 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652369 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652387 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652389 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652392 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652394 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652397 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652400 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652402 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652405 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652407 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652410 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652412 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652415 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652418 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 17:00:30.656303 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652420 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652422 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.652425 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652495 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652505 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652515 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652521 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652528 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652533 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652538 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652542 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652545 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652548 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652551 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652555 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652558 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652561 2576 flags.go:64] FLAG: --cgroup-root="" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652564 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652567 2576 flags.go:64] FLAG: --client-ca-file="" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652570 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652573 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652578 2576 flags.go:64] FLAG: --cluster-domain="" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652581 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652584 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 17:00:30.656797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652587 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652590 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652593 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652596 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652599 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652602 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652605 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652608 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652611 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652614 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652617 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652621 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652624 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652626 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652630 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652633 2576 flags.go:64] FLAG: --enable-server="true" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652636 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652640 2576 flags.go:64] FLAG: --event-burst="100" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652643 2576 flags.go:64] FLAG: --event-qps="50" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652646 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652649 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652652 2576 flags.go:64] FLAG: --eviction-hard="" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652661 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652664 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652667 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 17:00:30.657395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652670 2576 flags.go:64] FLAG: --eviction-soft="" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652673 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652676 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652678 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652682 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652685 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652688 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652691 2576 flags.go:64] FLAG: --feature-gates="" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652694 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652697 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652700 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652703 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652706 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652709 2576 flags.go:64] FLAG: --help="false" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652711 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-130-254.ec2.internal" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652714 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652717 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652720 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652723 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652726 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652728 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652735 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652738 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 17:00:30.658027 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652741 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652744 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652747 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652750 2576 flags.go:64] FLAG: --kube-reserved="" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652752 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652763 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652766 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652769 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652772 2576 flags.go:64] FLAG: --lock-file="" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652775 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652777 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652781 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652786 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652788 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652792 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652795 2576 flags.go:64] FLAG: --logging-format="text" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652798 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652801 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652804 2576 flags.go:64] FLAG: --manifest-url="" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652807 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652811 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652814 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652818 2576 flags.go:64] FLAG: --max-pods="110" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652821 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652823 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 17:00:30.658645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652826 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652829 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652832 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652834 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652837 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652849 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652852 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652855 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652858 2576 flags.go:64] FLAG: --pod-cidr="" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652861 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652866 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652869 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652872 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652874 2576 flags.go:64] FLAG: --port="10250" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652877 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652880 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-099087c8de9c7b7e3" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652883 2576 flags.go:64] FLAG: --qos-reserved="" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652887 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652890 2576 flags.go:64] FLAG: --register-node="true" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652892 2576 flags.go:64] FLAG: --register-with-taints="" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652901 2576 flags.go:64] FLAG: --registry-burst="10" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652904 2576 flags.go:64] FLAG: --registry-qps="5" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652907 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652910 2576 flags.go:64] FLAG: --reserved-memory="" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652916 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 17:00:30.659654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652919 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652921 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652924 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652927 2576 flags.go:64] FLAG: --runonce="false" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652930 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652933 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652936 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652939 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652942 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652945 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652948 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652951 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652955 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652959 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652962 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652971 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652975 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652978 2576 flags.go:64] FLAG: --system-cgroups="" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652981 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652986 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652989 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652992 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652996 2576 flags.go:64] FLAG: --tls-min-version="" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.652999 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653001 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 17:00:30.660695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653023 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653026 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653029 2576 flags.go:64] FLAG: --v="2" Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653033 2576 flags.go:64] FLAG: --version="false" Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653037 2576 flags.go:64] FLAG: --vmodule="" Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653041 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653045 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653160 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653164 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653168 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653170 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653174 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653177 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653179 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653182 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653184 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653187 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653190 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653193 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653197 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653199 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 17:00:30.661512 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653202 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653205 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653207 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653210 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653213 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653216 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653218 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653221 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653223 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653226 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653228 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653231 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653235 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653238 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653241 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653243 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653246 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653249 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653252 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 17:00:30.662120 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653254 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653257 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653259 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653261 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653264 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653267 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653269 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653272 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653274 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653277 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653279 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653284 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653287 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653289 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653292 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653294 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653297 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653299 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653302 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653304 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 17:00:30.662970 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653307 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653309 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653312 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653314 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653317 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653320 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653322 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653324 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653327 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653329 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653332 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653335 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653337 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653340 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653342 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653345 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653347 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653350 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653352 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653355 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 17:00:30.663684 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653357 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653360 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653362 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653366 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653369 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653384 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653387 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653390 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653392 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653395 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653397 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653400 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653403 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653406 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653410 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653413 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653416 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653419 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653422 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 17:00:30.664347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653424 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653428 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653430 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653433 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653435 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653439 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.653442 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.653450 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.660990 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.661026 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.661923 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.661937 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662021 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662029 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662035 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 17:00:30.664900 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662039 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662045 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662049 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662054 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662058 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662063 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662070 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662076 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662082 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662086 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662091 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662096 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662100 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662104 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662108 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662113 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662117 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662123 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662128 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 17:00:30.665299 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662133 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662137 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662141 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662145 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662149 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662154 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662159 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662166 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662170 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662175 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662179 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662183 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662199 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662203 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662207 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662211 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662215 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662219 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662224 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 17:00:30.666222 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662228 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662234 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662240 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662245 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662249 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662254 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662258 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662263 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662267 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662271 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662275 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662279 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662283 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662287 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662291 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662296 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662300 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662304 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662308 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 17:00:30.666833 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662313 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662317 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662322 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662326 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662331 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662335 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662340 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662343 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662348 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662351 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662356 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662360 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662364 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662368 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662387 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662391 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662396 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662400 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662404 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662408 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 17:00:30.667423 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662412 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662416 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662420 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662424 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662428 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662432 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662436 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662440 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662444 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662448 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662452 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662457 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662461 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662466 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662470 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662475 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662480 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662484 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 17:00:30.667901 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662488 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.662497 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662700 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662708 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662714 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662718 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662722 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662727 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662732 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662736 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662740 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662745 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662749 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662753 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662758 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 17:00:30.668347 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662762 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662767 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662772 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662777 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662785 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662791 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662797 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662802 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662807 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662811 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662816 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662821 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662826 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662831 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662835 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662840 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662845 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662849 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662853 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662858 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 17:00:30.668719 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662862 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662867 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662871 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662875 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662879 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662883 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662887 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662891 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662894 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662899 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662903 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662907 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662911 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662915 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662919 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662923 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662927 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662931 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662936 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662940 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 17:00:30.669283 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662944 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662948 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662952 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662956 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662962 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662966 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662970 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662974 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662979 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662984 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662987 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662991 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662995 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.662999 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663004 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663008 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663012 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663016 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663020 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663024 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 17:00:30.669875 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663028 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663032 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663036 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663040 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663044 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663048 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663052 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663056 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663060 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663064 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663068 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663072 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663076 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663080 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663084 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663088 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663095 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663101 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663105 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 17:00:30.670359 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663109 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663113 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663118 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663122 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663126 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663130 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:30.663134 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.663142 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.663303 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.667056 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.669027 2576 server.go:1019] "Starting client certificate rotation" Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.669128 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 17:00:30.670853 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.669168 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 17:00:30.671645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.671629 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 17:00:30.672735 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.672717 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 17:00:30.679157 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.679142 2576 log.go:25] "Validated CRI v1 runtime API" Jul 30 17:00:30.682447 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.682434 2576 log.go:25] "Validated CRI v1 image API" Jul 30 17:00:30.683144 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.683129 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 17:00:30.686715 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.686696 2576 fs.go:135] Filesystem UUIDs: map[1f07268a-e240-4f19-a3aa-d0ade1a6c8ec:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d68c4801-9a81-4e0f-be99-646c94e94dea:/dev/nvme0n1p4] Jul 30 17:00:30.686781 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.686714 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 17:00:30.692532 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.692422 2576 manager.go:216] Machine: {Timestamp:2026-07-30 17:00:30.691075812 +0000 UTC m=+0.364273280 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104575 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269ed07f4ec1db4a05466ea79f435a SystemUUID:ec269ed0-7f4e-c1db-4a05-466ea79f435a BootID:2587b801-b1ea-41e7-9832-9927132a3033 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ea:dd:61:7b:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ea:dd:61:7b:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:2f:36:23:d4:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 17:00:30.692532 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.692529 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 17:00:30.692637 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.692608 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 17:00:30.695396 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.695360 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 17:00:30.695538 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.695398 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-254.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 17:00:30.696644 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.696635 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 17:00:30.696684 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.696647 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 17:00:30.696684 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.696655 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 17:00:30.697153 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.697143 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 17:00:30.697965 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.697954 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 17:00:30.699757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.699746 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 30 17:00:30.699874 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.699864 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 17:00:30.700575 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.700564 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 30 17:00:30.700614 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.700579 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 17:00:30.700614 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.700590 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 17:00:30.700614 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.700599 2576 kubelet.go:398] "Adding apiserver pod source" Jul 30 17:00:30.700614 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.700609 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 17:00:30.701106 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.701088 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 17:00:30.701166 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.701115 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 17:00:30.702803 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.702784 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 17:00:30.703220 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.703204 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 17:00:30.705108 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.705095 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 17:00:30.705162 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.705125 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 17:00:30.706589 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706577 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706595 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706601 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706607 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706617 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706623 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706630 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706636 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706643 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 17:00:30.706646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706648 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 17:00:30.706879 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706657 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 17:00:30.706879 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.706662 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 17:00:30.707600 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.707590 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 17:00:30.707600 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.707600 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 17:00:30.710704 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.710692 2576 server.go:1268] "Started kubelet" Jul 30 17:00:30.710783 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.710754 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 17:00:30.710846 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.710772 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 17:00:30.714920 ip-10-0-130-254 systemd[1]: Started Kubernetes Kubelet. Jul 30 17:00:30.715235 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.715209 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 17:00:30.715659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.715549 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-254.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 17:00:30.716175 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.716044 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 17:00:30.716175 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.715911 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 17:00:30.716725 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.716704 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 30 17:00:30.719710 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.719690 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 17:00:30.720147 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.720130 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 17:00:30.720907 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.719904 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-254.ec2.internal.18c71fe1838a309d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-254.ec2.internal,UID:ip-10-0-130-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-254.ec2.internal,},FirstTimestamp:2026-07-30 17:00:30.710665373 +0000 UTC m=+0.383862842,LastTimestamp:2026-07-30 17:00:30.710665373 +0000 UTC m=+0.383862842,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-254.ec2.internal,}" Jul 30 17:00:30.725865 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.725847 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 17:00:30.725865 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.725854 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 17:00:30.726001 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.725925 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:30.726505 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726488 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 17:00:30.726637 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726554 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 17:00:30.726745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726640 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 17:00:30.726745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726654 2576 factory.go:55] Registering systemd factory Jul 30 17:00:30.726745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726667 2576 factory.go:223] Registration of the systemd container factory successfully Jul 30 17:00:30.726879 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726749 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 30 17:00:30.726879 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726758 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 30 17:00:30.726959 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726914 2576 factory.go:153] Registering CRI-O factory Jul 30 17:00:30.726959 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726926 2576 factory.go:223] Registration of the crio container factory successfully Jul 30 17:00:30.727026 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726967 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 17:00:30.727026 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726985 2576 factory.go:103] Registering Raw factory Jul 30 17:00:30.727026 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.726996 2576 manager.go:1196] Started watching for new ooms in manager Jul 30 17:00:30.727313 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.727302 2576 manager.go:349] Starting recovery of all containers Jul 30 17:00:30.728902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.728879 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6crcz" Jul 30 17:00:30.735087 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.735065 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6crcz" Jul 30 17:00:30.735463 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.735434 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 17:00:30.735579 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.735545 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 17:00:30.737892 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.737874 2576 manager.go:354] Recovery completed Jul 30 17:00:30.742704 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.742689 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 17:00:30.744432 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.744418 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientMemory" Jul 30 17:00:30.744500 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.744444 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 17:00:30.744500 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.744454 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientPID" Jul 30 17:00:30.744893 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.744880 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 17:00:30.744893 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.744892 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 17:00:30.745004 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.744905 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 17:00:30.745004 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.744907 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 30 17:00:30.745591 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.745576 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientMemory" Jul 30 17:00:30.745632 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.745602 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 17:00:30.745632 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.745615 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientPID" Jul 30 17:00:30.746629 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.746603 2576 policy_none.go:49] "None policy: Start" Jul 30 17:00:30.746629 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.746617 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 17:00:30.746629 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.746572 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-254.ec2.internal.18c71fe1858d6b86 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-254.ec2.internal,UID:ip-10-0-130-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-254.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-254.ec2.internal,},FirstTimestamp:2026-07-30 17:00:30.744431494 +0000 UTC m=+0.417628964,LastTimestamp:2026-07-30 17:00:30.744431494 +0000 UTC m=+0.417628964,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-254.ec2.internal,}" Jul 30 17:00:30.746807 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.746690 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 17:00:30.750007 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.749989 2576 policy_none.go:47] "Start" Jul 30 17:00:30.789628 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.789507 2576 manager.go:337] "Starting Device Plugin manager" Jul 30 17:00:30.789720 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.789650 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 17:00:30.789720 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.789663 2576 server.go:87] "Starting device plugin registration server" Jul 30 17:00:30.789920 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.789907 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 17:00:30.789982 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.789922 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 17:00:30.790029 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.789995 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 17:00:30.790080 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.790071 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 17:00:30.790080 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.790080 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 17:00:30.790607 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.790585 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 17:00:30.790673 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.790638 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:30.890616 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.890574 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 17:00:30.891899 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.891567 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientMemory" Jul 30 17:00:30.891899 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.891604 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 17:00:30.891899 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.891619 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientPID" Jul 30 17:00:30.891899 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.891645 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-254.ec2.internal" Jul 30 17:00:30.898150 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.898101 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-254.ec2.internal" Jul 30 17:00:30.898150 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.898136 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-254.ec2.internal\": node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:30.914066 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.914046 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:30.931572 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.931551 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 17:00:30.932724 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.932707 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 17:00:30.932795 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.932738 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 17:00:30.932795 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.932756 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 17:00:30.932795 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:30.932789 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 17:00:30.934789 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:30.934770 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 17:00:31.015246 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.015202 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:31.032918 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.032886 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal"] Jul 30 17:00:31.033019 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.032987 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 17:00:31.034684 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.034668 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientMemory" Jul 30 17:00:31.034764 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.034702 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 17:00:31.034764 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.034718 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientPID" Jul 30 17:00:31.036143 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.036129 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 17:00:31.036293 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.036267 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.036343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.036308 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 17:00:31.036930 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.036914 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientMemory" Jul 30 17:00:31.037012 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.036941 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 17:00:31.037012 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.036954 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientPID" Jul 30 17:00:31.037170 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.037024 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientMemory" Jul 30 17:00:31.037170 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.037059 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 17:00:31.037170 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.037074 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientPID" Jul 30 17:00:31.038418 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.038400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.038524 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.038434 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 17:00:31.039126 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.039104 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientMemory" Jul 30 17:00:31.039222 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.039136 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 17:00:31.039222 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.039147 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeHasSufficientPID" Jul 30 17:00:31.062144 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.062120 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-254.ec2.internal\" not found" node="ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.065741 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.065723 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-254.ec2.internal\" not found" node="ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.115316 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.115285 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:31.129306 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.129286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca7232da2ceebd3c2cd75f2c8e9f8e8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal\" (UID: \"ca7232da2ceebd3c2cd75f2c8e9f8e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.129365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.129311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7232da2ceebd3c2cd75f2c8e9f8e8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal\" (UID: \"ca7232da2ceebd3c2cd75f2c8e9f8e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.129365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.129329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55e30e52bbfc0cb8ac949e33c9db102a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-254.ec2.internal\" (UID: \"55e30e52bbfc0cb8ac949e33c9db102a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.215871 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.215801 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:31.230168 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.230145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca7232da2ceebd3c2cd75f2c8e9f8e8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal\" (UID: \"ca7232da2ceebd3c2cd75f2c8e9f8e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.230238 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.230172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7232da2ceebd3c2cd75f2c8e9f8e8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal\" (UID: \"ca7232da2ceebd3c2cd75f2c8e9f8e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.230287 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.230250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca7232da2ceebd3c2cd75f2c8e9f8e8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal\" (UID: \"ca7232da2ceebd3c2cd75f2c8e9f8e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.230287 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.230277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55e30e52bbfc0cb8ac949e33c9db102a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-254.ec2.internal\" (UID: \"55e30e52bbfc0cb8ac949e33c9db102a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.230353 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.230229 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca7232da2ceebd3c2cd75f2c8e9f8e8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal\" (UID: \"ca7232da2ceebd3c2cd75f2c8e9f8e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.230353 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.230331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55e30e52bbfc0cb8ac949e33c9db102a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-254.ec2.internal\" (UID: \"55e30e52bbfc0cb8ac949e33c9db102a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.322387 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.322339 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:31.367217 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.367191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.369519 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.369502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.423353 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.423318 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:31.528089 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.528017 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:31.628767 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:31.628740 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-254.ec2.internal\" not found" Jul 30 17:00:31.668803 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.668770 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 17:00:31.669248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.668956 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 17:00:31.702653 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.702629 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 17:00:31.725991 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.725964 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.725991 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.725990 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 17:00:31.737464 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.737443 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 17:00:31.739536 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.739521 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" Jul 30 17:00:31.739906 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.739631 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 16:55:30 +0000 UTC" deadline="2028-05-13 08:19:49.110595154 +0000 UTC" Jul 30 17:00:31.739906 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.739666 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15663h19m17.370931332s" Jul 30 17:00:31.739906 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.739812 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 17:00:31.748948 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.748927 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 17:00:31.762111 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.762093 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 17:00:31.762338 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.762323 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtc64" Jul 30 17:00:31.770676 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:31.770655 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtc64" Jul 30 17:00:32.224355 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.224329 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 17:00:32.315242 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:32.315209 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e30e52bbfc0cb8ac949e33c9db102a.slice/crio-02f2283f81c709f332fd5e4c928f1a37ba3a939e0802af1f943aae9a97abebb5 WatchSource:0}: Error finding container 02f2283f81c709f332fd5e4c928f1a37ba3a939e0802af1f943aae9a97abebb5: Status 404 returned error can't find the container with id 02f2283f81c709f332fd5e4c928f1a37ba3a939e0802af1f943aae9a97abebb5 Jul 30 17:00:32.320240 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.320223 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 17:00:32.485503 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:32.485474 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7232da2ceebd3c2cd75f2c8e9f8e8f.slice/crio-bca4be97d1d18620f2ca401f6f0f2e23d885bce43a2c95b1601c872f39d15704 WatchSource:0}: Error finding container bca4be97d1d18620f2ca401f6f0f2e23d885bce43a2c95b1601c872f39d15704: Status 404 returned error can't find the container with id bca4be97d1d18620f2ca401f6f0f2e23d885bce43a2c95b1601c872f39d15704 Jul 30 17:00:32.703513 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.703477 2576 apiserver.go:52] "Watching apiserver" Jul 30 17:00:32.711762 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.711737 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 17:00:32.712132 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.712110 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-p8znm","kube-system/global-pull-secret-syncer-9wmwb","kube-system/konnectivity-agent-9hzs9","kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz","openshift-cluster-node-tuning-operator/tuned-ghlw7","openshift-image-registry/node-ca-v4kvx","openshift-multus/multus-qxth4","openshift-multus/network-metrics-daemon-jt82f","openshift-dns/node-resolver-rfrvd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal","openshift-multus/multus-additional-cni-plugins-lzck8","openshift-network-diagnostics/network-check-target-rsvsz","openshift-network-operator/iptables-alerter-vv5k6"] Jul 30 17:00:32.713757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.713738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.715100 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.715078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.715204 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.715143 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:32.716479 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.716453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:32.716716 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.716696 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 17:00:32.716716 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.716696 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 17:00:32.717447 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.717418 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 17:00:32.717537 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.717448 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 17:00:32.717537 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.717519 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nkpv9\"" Jul 30 17:00:32.717652 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.717523 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 17:00:32.717652 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.717581 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 17:00:32.717877 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.717861 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.719084 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.719062 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 17:00:32.719221 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.719167 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 17:00:32.719221 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.719168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.719371 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.719069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ndzx5\"" Jul 30 17:00:32.719886 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.719867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 17:00:32.720494 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.720454 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4mspj\"" Jul 30 17:00:32.720787 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.720768 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 17:00:32.720875 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.720825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 17:00:32.722715 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.722696 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 17:00:32.724258 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.723078 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 17:00:32.724258 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.723202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b4w4b\"" Jul 30 17:00:32.724258 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.723435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.725683 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.725666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.726196 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.726165 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 17:00:32.726309 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.726295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z94lz\"" Jul 30 17:00:32.726591 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.726577 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 17:00:32.726722 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.726710 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 17:00:32.727194 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.727176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:32.727267 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.727250 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:32.728881 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.728855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.729917 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.729889 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 17:00:32.730026 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.729921 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 17:00:32.730026 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.729892 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 17:00:32.730146 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.730067 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 17:00:32.730318 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.730297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.730318 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.730313 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-728cp\"" Jul 30 17:00:32.731101 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.731058 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 17:00:32.731101 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.731075 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 17:00:32.731421 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.731406 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6n7g5\"" Jul 30 17:00:32.731606 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.731590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:32.731672 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.731655 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:32.732485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.732467 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 17:00:32.732804 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.732785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fmppd\"" Jul 30 17:00:32.733150 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.733137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 17:00:32.733288 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.733247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.735487 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.735469 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 17:00:32.735650 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.735631 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k7r7w\"" Jul 30 17:00:32.735734 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.735660 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 17:00:32.735734 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.735698 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 17:00:32.739006 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.738990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-log-socket\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-os-release\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-daemon-config\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-systemd\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovnkube-config\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739272 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovnkube-script-lib\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739272 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-cnibin\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739272 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-cni-binary-copy\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739272 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-host\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.739272 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-cni-netd\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739272 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3c11f0-4a5b-4291-9b07-daaae470ee87-tmp\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.739480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-env-overrides\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-netns\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-etc-kubernetes\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8l9h\" (UniqueName: \"kubernetes.io/projected/6bd1a463-eb1a-4afb-aba9-ea8294291672-kube-api-access-p8l9h\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:32.739480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-socket-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.739480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-registration-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.739659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-socket-dir-parent\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-cni-bin\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-var-lib-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovn-node-metrics-cert\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswbp\" (UniqueName: \"kubernetes.io/projected/85ef8995-7981-46e0-b9b0-b0b7b4234643-kube-api-access-mswbp\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-multus-certs\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-kubernetes\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysctl-d\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-etc-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-hostroot\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrcr\" (UniqueName: \"kubernetes.io/projected/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-kube-api-access-lwrcr\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-var-lib-kubelet\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysconfig\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.739868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-kubelet\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8eb802f4-7b82-453c-8997-d9ff1248192c-agent-certs\") pod \"konnectivity-agent-9hzs9\" (UID: \"8eb802f4-7b82-453c-8997-d9ff1248192c\") " pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-kubelet\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.739992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-cni-bin\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-conf-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-device-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.740103 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-systemd\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-sys\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-host\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-cni-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-k8s-cni-cncf-io\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-cni-multus\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-sys-fs\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hd4\" (UniqueName: \"kubernetes.io/projected/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-kube-api-access-t6hd4\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-lib-modules\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btktj\" (UniqueName: \"kubernetes.io/projected/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-kube-api-access-btktj\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.740365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-run-netns\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/479c1d63-717e-42f7-b325-e3585799cd58-dbus\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-system-cni-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-modprobe-d\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysctl-conf\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-run\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-tuned\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-node-log\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-etc-selinux\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmk6\" (UniqueName: \"kubernetes.io/projected/9c3c11f0-4a5b-4291-9b07-daaae470ee87-kube-api-access-fgmk6\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-ovn\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-serviceca\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-systemd-units\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.740786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-slash\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.741384 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/479c1d63-717e-42f7-b325-e3585799cd58-kubelet-config\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.741384 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.740826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8eb802f4-7b82-453c-8997-d9ff1248192c-konnectivity-ca\") pod \"konnectivity-agent-9hzs9\" (UID: \"8eb802f4-7b82-453c-8997-d9ff1248192c\") " pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:32.772644 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.772589 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:31 +0000 UTC" deadline="2028-05-08 17:23:37.182567162 +0000 UTC" Jul 30 17:00:32.772644 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.772638 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15552h23m4.409934003s" Jul 30 17:00:32.827367 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.827342 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 17:00:32.841016 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.840994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-sys\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.841072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-host\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.841072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-cni-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-k8s-cni-cncf-io\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-cni-multus\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-sys-fs\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hd4\" (UniqueName: \"kubernetes.io/projected/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-kube-api-access-t6hd4\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-lib-modules\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841122 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-host\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-sys\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841165 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-cni-multus\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btktj\" (UniqueName: \"kubernetes.io/projected/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-kube-api-access-btktj\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-sys-fs\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.841248 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6l2q\" (UniqueName: \"kubernetes.io/projected/caf62b48-154a-4b0c-941b-506218889e2d-kube-api-access-g6l2q\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/caf62b48-154a-4b0c-941b-506218889e2d-hosts-file\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-k8s-cni-cncf-io\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-run-netns\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-lib-modules\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/479c1d63-717e-42f7-b325-e3585799cd58-dbus\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-run-netns\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-iptables-alerter-script\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-cni-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/479c1d63-717e-42f7-b325-e3585799cd58-dbus\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-system-cni-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-system-cni-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-modprobe-d\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysctl-conf\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-run\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.841674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-tuned\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-host-slash\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-node-log\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-modprobe-d\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-run\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-node-log\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysctl-conf\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-etc-selinux\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmk6\" (UniqueName: \"kubernetes.io/projected/9c3c11f0-4a5b-4291-9b07-daaae470ee87-kube-api-access-fgmk6\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.841823 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf62b48-154a-4b0c-941b-506218889e2d-tmp-dir\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841874 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-etc-selinux\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.841891 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret podName:479c1d63-717e-42f7-b325-e3585799cd58 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:33.341867996 +0000 UTC m=+3.015065468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret") pod "global-pull-secret-syncer-9wmwb" (UID: "479c1d63-717e-42f7-b325-e3585799cd58") : object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-os-release\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-ovn\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.842581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841967 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.841985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-serviceca\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-systemd-units\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-ovn\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-slash\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/479c1d63-717e-42f7-b325-e3585799cd58-kubelet-config\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-systemd-units\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8eb802f4-7b82-453c-8997-d9ff1248192c-konnectivity-ca\") pod \"konnectivity-agent-9hzs9\" (UID: \"8eb802f4-7b82-453c-8997-d9ff1248192c\") " pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-slash\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-kube-api-access-2794j\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-log-socket\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/479c1d63-717e-42f7-b325-e3585799cd58-kubelet-config\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-os-release\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-daemon-config\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.843495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-log-socket\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-systemd\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovnkube-config\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-os-release\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovnkube-script-lib\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-cnibin\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-cni-binary-copy\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-systemd\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-host\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-cnibin\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-system-cni-dir\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-cni-netd\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-serviceca\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3c11f0-4a5b-4291-9b07-daaae470ee87-tmp\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-env-overrides\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-netns\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-etc-kubernetes\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8l9h\" (UniqueName: \"kubernetes.io/projected/6bd1a463-eb1a-4afb-aba9-ea8294291672-kube-api-access-p8l9h\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:32.844343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-socket-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-registration-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-socket-dir-parent\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-cni-bin\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8eb802f4-7b82-453c-8997-d9ff1248192c-konnectivity-ca\") pod \"konnectivity-agent-9hzs9\" (UID: \"8eb802f4-7b82-453c-8997-d9ff1248192c\") " pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cnibin\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-cni-netd\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-var-lib-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-host\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovn-node-metrics-cert\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswbp\" (UniqueName: \"kubernetes.io/projected/85ef8995-7981-46e0-b9b0-b0b7b4234643-kube-api-access-mswbp\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-cni-bin\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-multus-certs\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843002 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-socket-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-kubernetes\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.845304 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-kubernetes\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovnkube-script-lib\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-registration-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysctl-d\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843104 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovnkube-config\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcwf\" (UniqueName: \"kubernetes.io/projected/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-kube-api-access-hdcwf\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-cni-binary-copy\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843151 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-socket-dir-parent\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysctl-d\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-etc-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-netns\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.842902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-daemon-config\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-etc-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843242 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-run-multus-certs\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-hostroot\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrcr\" (UniqueName: \"kubernetes.io/projected/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-kube-api-access-lwrcr\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-hostroot\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-etc-kubernetes\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-var-lib-kubelet\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-var-lib-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysconfig\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-kubelet\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8eb802f4-7b82-453c-8997-d9ff1248192c-agent-certs\") pod \"konnectivity-agent-9hzs9\" (UID: \"8eb802f4-7b82-453c-8997-d9ff1248192c\") " pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-kubelet\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-var-lib-kubelet\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/85ef8995-7981-46e0-b9b0-b0b7b4234643-env-overrides\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-sysconfig\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-kubelet\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.843795 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-host-var-lib-kubelet\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.843868 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs podName:6bd1a463-eb1a-4afb-aba9-ea8294291672 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:33.343855816 +0000 UTC m=+3.017053277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs") pod "network-metrics-daemon-jt82f" (UID: "6bd1a463-eb1a-4afb-aba9-ea8294291672") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.846806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-run-openvswitch\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-cni-bin\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-run-ovn-kubernetes\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.843982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-conf-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.844000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-device-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.844046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-device-dir\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.844050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-multus-conf-dir\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.844056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-systemd\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.844095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85ef8995-7981-46e0-b9b0-b0b7b4234643-host-cni-bin\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.844148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-systemd\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.846086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c3c11f0-4a5b-4291-9b07-daaae470ee87-etc-tuned\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.846140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3c11f0-4a5b-4291-9b07-daaae470ee87-tmp\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.846184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/85ef8995-7981-46e0-b9b0-b0b7b4234643-ovn-node-metrics-cert\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.847333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.846265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8eb802f4-7b82-453c-8997-d9ff1248192c-agent-certs\") pod \"konnectivity-agent-9hzs9\" (UID: \"8eb802f4-7b82-453c-8997-d9ff1248192c\") " pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:32.852831 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.852808 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hd4\" (UniqueName: \"kubernetes.io/projected/c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18-kube-api-access-t6hd4\") pod \"aws-ebs-csi-driver-node-45gfz\" (UID: \"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:32.853332 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.853307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btktj\" (UniqueName: \"kubernetes.io/projected/78e6945a-53cb-4b71-af2e-b91cdfdbcb90-kube-api-access-btktj\") pod \"node-ca-v4kvx\" (UID: \"78e6945a-53cb-4b71-af2e-b91cdfdbcb90\") " pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:32.853565 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.853547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmk6\" (UniqueName: \"kubernetes.io/projected/9c3c11f0-4a5b-4291-9b07-daaae470ee87-kube-api-access-fgmk6\") pod \"tuned-ghlw7\" (UID: \"9c3c11f0-4a5b-4291-9b07-daaae470ee87\") " pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:32.853753 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.853737 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrcr\" (UniqueName: \"kubernetes.io/projected/063d41b9-4280-4bfa-9eed-4ce6ef4084d3-kube-api-access-lwrcr\") pod \"multus-qxth4\" (UID: \"063d41b9-4280-4bfa-9eed-4ce6ef4084d3\") " pod="openshift-multus/multus-qxth4" Jul 30 17:00:32.854975 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.854952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswbp\" (UniqueName: \"kubernetes.io/projected/85ef8995-7981-46e0-b9b0-b0b7b4234643-kube-api-access-mswbp\") pod \"ovnkube-node-p8znm\" (UID: \"85ef8995-7981-46e0-b9b0-b0b7b4234643\") " pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:32.855053 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.854956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8l9h\" (UniqueName: \"kubernetes.io/projected/6bd1a463-eb1a-4afb-aba9-ea8294291672-kube-api-access-p8l9h\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:32.936834 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.936797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" event={"ID":"ca7232da2ceebd3c2cd75f2c8e9f8e8f","Type":"ContainerStarted","Data":"bca4be97d1d18620f2ca401f6f0f2e23d885bce43a2c95b1601c872f39d15704"} Jul 30 17:00:32.937852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.937833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" event={"ID":"55e30e52bbfc0cb8ac949e33c9db102a","Type":"ContainerStarted","Data":"02f2283f81c709f332fd5e4c928f1a37ba3a939e0802af1f943aae9a97abebb5"} Jul 30 17:00:32.944523 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.944581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcwf\" (UniqueName: \"kubernetes.io/projected/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-kube-api-access-hdcwf\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.944581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6l2q\" (UniqueName: \"kubernetes.io/projected/caf62b48-154a-4b0c-941b-506218889e2d-kube-api-access-g6l2q\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.944660 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/caf62b48-154a-4b0c-941b-506218889e2d-hosts-file\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.944660 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.944750 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-iptables-alerter-script\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.944750 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/caf62b48-154a-4b0c-941b-506218889e2d-hosts-file\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.944846 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.944898 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-host-slash\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.944898 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf62b48-154a-4b0c-941b-506218889e2d-tmp-dir\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.944997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:32.944997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-os-release\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.944997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-host-slash\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.944997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-kube-api-access-2794j\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945137 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.944995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945137 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-system-cni-dir\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945137 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945137 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cnibin\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945137 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-os-release\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945137 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cnibin\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945352 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-system-cni-dir\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945352 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945232 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945352 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf62b48-154a-4b0c-941b-506218889e2d-tmp-dir\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.945352 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-iptables-alerter-script\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.945778 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.945864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.945847 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:32.951685 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.951666 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 17:00:32.951685 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.951686 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 17:00:32.951843 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.951695 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2b8b for pod openshift-network-diagnostics/network-check-target-rsvsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:32.951843 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:32.951737 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b podName:47edcc66-8e41-41b6-9bb3-c457deebbda9 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:33.451725481 +0000 UTC m=+3.124922959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x2b8b" (UniqueName: "kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b") pod "network-check-target-rsvsz" (UID: "47edcc66-8e41-41b6-9bb3-c457deebbda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:32.953139 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.953116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcwf\" (UniqueName: \"kubernetes.io/projected/d78dd639-1eea-42cb-9b55-b13ebcdbe59d-kube-api-access-hdcwf\") pod \"iptables-alerter-vv5k6\" (UID: \"d78dd639-1eea-42cb-9b55-b13ebcdbe59d\") " pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:32.953557 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.953541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6l2q\" (UniqueName: \"kubernetes.io/projected/caf62b48-154a-4b0c-941b-506218889e2d-kube-api-access-g6l2q\") pod \"node-resolver-rfrvd\" (UID: \"caf62b48-154a-4b0c-941b-506218889e2d\") " pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:32.953893 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:32.953873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2794j\" (UniqueName: \"kubernetes.io/projected/db4dcb51-1f61-41d0-9f81-d5b71f4e1c49-kube-api-access-2794j\") pod \"multus-additional-cni-plugins-lzck8\" (UID: \"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49\") " pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:33.025644 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.025591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:33.031611 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.031593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:33.034382 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.034359 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ef8995_7981_46e0_b9b0_b0b7b4234643.slice/crio-1164271ffc9fe2ab2e2b5d134335cdfff309b782890f448e167c1367e9b08d05 WatchSource:0}: Error finding container 1164271ffc9fe2ab2e2b5d134335cdfff309b782890f448e167c1367e9b08d05: Status 404 returned error can't find the container with id 1164271ffc9fe2ab2e2b5d134335cdfff309b782890f448e167c1367e9b08d05 Jul 30 17:00:33.039107 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.039078 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb802f4_7b82_453c_8997_d9ff1248192c.slice/crio-d42b3ee7bc3f0607bce16b12ab724247e085a3a50329454ab8143e2f0daa2f35 WatchSource:0}: Error finding container d42b3ee7bc3f0607bce16b12ab724247e085a3a50329454ab8143e2f0daa2f35: Status 404 returned error can't find the container with id d42b3ee7bc3f0607bce16b12ab724247e085a3a50329454ab8143e2f0daa2f35 Jul 30 17:00:33.039844 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.039830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" Jul 30 17:00:33.045013 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.044995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" Jul 30 17:00:33.048873 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.048852 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94e5c6c_b5d3_4bb3_b52a_a6cd7dd3ed18.slice/crio-8e248cd8c0caa9bd90022fe159daf733f1d885fa0dc0d6b584a7845fab7266b3 WatchSource:0}: Error finding container 8e248cd8c0caa9bd90022fe159daf733f1d885fa0dc0d6b584a7845fab7266b3: Status 404 returned error can't find the container with id 8e248cd8c0caa9bd90022fe159daf733f1d885fa0dc0d6b584a7845fab7266b3 Jul 30 17:00:33.050176 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.050102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v4kvx" Jul 30 17:00:33.053546 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.053475 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 17:00:33.053694 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.053670 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3c11f0_4a5b_4291_9b07_daaae470ee87.slice/crio-29ce62e90b7edc1897e30a5e6a7dc7b6112c0f87634a4ff646ee3b9b928043da WatchSource:0}: Error finding container 29ce62e90b7edc1897e30a5e6a7dc7b6112c0f87634a4ff646ee3b9b928043da: Status 404 returned error can't find the container with id 29ce62e90b7edc1897e30a5e6a7dc7b6112c0f87634a4ff646ee3b9b928043da Jul 30 17:00:33.055960 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.055942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qxth4" Jul 30 17:00:33.059216 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.059193 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e6945a_53cb_4b71_af2e_b91cdfdbcb90.slice/crio-f0d2366522de46be0a76fa44e24ae5e301c750ac0567acb72851aa25684e0cd3 WatchSource:0}: Error finding container f0d2366522de46be0a76fa44e24ae5e301c750ac0567acb72851aa25684e0cd3: Status 404 returned error can't find the container with id f0d2366522de46be0a76fa44e24ae5e301c750ac0567acb72851aa25684e0cd3 Jul 30 17:00:33.062582 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.062507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfrvd" Jul 30 17:00:33.063454 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.063431 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063d41b9_4280_4bfa_9eed_4ce6ef4084d3.slice/crio-75080d29d3d17bcb0ffd5281d7593ec18cf202e9d2afad5d5d05cedb24bf261e WatchSource:0}: Error finding container 75080d29d3d17bcb0ffd5281d7593ec18cf202e9d2afad5d5d05cedb24bf261e: Status 404 returned error can't find the container with id 75080d29d3d17bcb0ffd5281d7593ec18cf202e9d2afad5d5d05cedb24bf261e Jul 30 17:00:33.067640 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.067623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzck8" Jul 30 17:00:33.071128 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.071110 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf62b48_154a_4b0c_941b_506218889e2d.slice/crio-23d83c655ef29c5d8fccc5cc364b9a1f17ceea6e6be66f3c09b08a2788aa53c2 WatchSource:0}: Error finding container 23d83c655ef29c5d8fccc5cc364b9a1f17ceea6e6be66f3c09b08a2788aa53c2: Status 404 returned error can't find the container with id 23d83c655ef29c5d8fccc5cc364b9a1f17ceea6e6be66f3c09b08a2788aa53c2 Jul 30 17:00:33.073213 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.073195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vv5k6" Jul 30 17:00:33.076969 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.076948 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4dcb51_1f61_41d0_9f81_d5b71f4e1c49.slice/crio-284a28ddc103294dea5e2d64ff285e1b352b5dce3bc326b4a596baf08e7b01f9 WatchSource:0}: Error finding container 284a28ddc103294dea5e2d64ff285e1b352b5dce3bc326b4a596baf08e7b01f9: Status 404 returned error can't find the container with id 284a28ddc103294dea5e2d64ff285e1b352b5dce3bc326b4a596baf08e7b01f9 Jul 30 17:00:33.082032 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:00:33.082009 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78dd639_1eea_42cb_9b55_b13ebcdbe59d.slice/crio-1fbac8713388115f9da1fd92b2328e7f1f54508b970e9a0062f7099de59d4329 WatchSource:0}: Error finding container 1fbac8713388115f9da1fd92b2328e7f1f54508b970e9a0062f7099de59d4329: Status 404 returned error can't find the container with id 1fbac8713388115f9da1fd92b2328e7f1f54508b970e9a0062f7099de59d4329 Jul 30 17:00:33.348861 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.348784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:33.348861 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.348855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:33.349149 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.348966 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:33.349149 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.348989 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:33.349149 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.349046 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs podName:6bd1a463-eb1a-4afb-aba9-ea8294291672 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:34.349025034 +0000 UTC m=+4.022222518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs") pod "network-metrics-daemon-jt82f" (UID: "6bd1a463-eb1a-4afb-aba9-ea8294291672") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:33.349149 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.349068 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret podName:479c1d63-717e-42f7-b325-e3585799cd58 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:34.349058324 +0000 UTC m=+4.022255791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret") pod "global-pull-secret-syncer-9wmwb" (UID: "479c1d63-717e-42f7-b325-e3585799cd58") : object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:33.551220 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.551178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:33.551408 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.551357 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 17:00:33.551408 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.551384 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 17:00:33.551408 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.551403 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2b8b for pod openshift-network-diagnostics/network-check-target-rsvsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:33.551585 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.551485 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b podName:47edcc66-8e41-41b6-9bb3-c457deebbda9 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:34.55146442 +0000 UTC m=+4.224661881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2b8b" (UniqueName: "kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b") pod "network-check-target-rsvsz" (UID: "47edcc66-8e41-41b6-9bb3-c457deebbda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:33.773852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.773729 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:31 +0000 UTC" deadline="2028-02-20 11:18:05.277593242 +0000 UTC" Jul 30 17:00:33.773852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.773801 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13674h17m31.503797214s" Jul 30 17:00:33.933104 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.933068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:33.933279 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.933215 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:33.933412 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.933388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:33.933533 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:33.933505 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:33.950170 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.950132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" event={"ID":"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18","Type":"ContainerStarted","Data":"8e248cd8c0caa9bd90022fe159daf733f1d885fa0dc0d6b584a7845fab7266b3"} Jul 30 17:00:33.957554 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.957525 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9hzs9" event={"ID":"8eb802f4-7b82-453c-8997-d9ff1248192c","Type":"ContainerStarted","Data":"d42b3ee7bc3f0607bce16b12ab724247e085a3a50329454ab8143e2f0daa2f35"} Jul 30 17:00:33.968874 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.968847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"1164271ffc9fe2ab2e2b5d134335cdfff309b782890f448e167c1367e9b08d05"} Jul 30 17:00:33.981544 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.981515 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vv5k6" event={"ID":"d78dd639-1eea-42cb-9b55-b13ebcdbe59d","Type":"ContainerStarted","Data":"1fbac8713388115f9da1fd92b2328e7f1f54508b970e9a0062f7099de59d4329"} Jul 30 17:00:33.993931 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.993868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfrvd" event={"ID":"caf62b48-154a-4b0c-941b-506218889e2d","Type":"ContainerStarted","Data":"23d83c655ef29c5d8fccc5cc364b9a1f17ceea6e6be66f3c09b08a2788aa53c2"} Jul 30 17:00:33.997054 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:33.997019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v4kvx" event={"ID":"78e6945a-53cb-4b71-af2e-b91cdfdbcb90","Type":"ContainerStarted","Data":"f0d2366522de46be0a76fa44e24ae5e301c750ac0567acb72851aa25684e0cd3"} Jul 30 17:00:34.026509 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:34.026388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" event={"ID":"9c3c11f0-4a5b-4291-9b07-daaae470ee87","Type":"ContainerStarted","Data":"29ce62e90b7edc1897e30a5e6a7dc7b6112c0f87634a4ff646ee3b9b928043da"} Jul 30 17:00:34.041322 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:34.041273 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerStarted","Data":"284a28ddc103294dea5e2d64ff285e1b352b5dce3bc326b4a596baf08e7b01f9"} Jul 30 17:00:34.043680 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:34.043627 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qxth4" event={"ID":"063d41b9-4280-4bfa-9eed-4ce6ef4084d3","Type":"ContainerStarted","Data":"75080d29d3d17bcb0ffd5281d7593ec18cf202e9d2afad5d5d05cedb24bf261e"} Jul 30 17:00:34.358454 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:34.358361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:34.358454 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:34.358436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:34.358667 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.358591 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:34.358667 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.358652 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret podName:479c1d63-717e-42f7-b325-e3585799cd58 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:36.35863158 +0000 UTC m=+6.031829057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret") pod "global-pull-secret-syncer-9wmwb" (UID: "479c1d63-717e-42f7-b325-e3585799cd58") : object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:34.359042 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.359024 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:34.359105 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.359077 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs podName:6bd1a463-eb1a-4afb-aba9-ea8294291672 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:36.359061608 +0000 UTC m=+6.032259077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs") pod "network-metrics-daemon-jt82f" (UID: "6bd1a463-eb1a-4afb-aba9-ea8294291672") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:34.560139 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:34.560104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:34.560347 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.560326 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 17:00:34.560431 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.560355 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 17:00:34.560431 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.560367 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2b8b for pod openshift-network-diagnostics/network-check-target-rsvsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:34.560545 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.560438 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b podName:47edcc66-8e41-41b6-9bb3-c457deebbda9 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:36.560418393 +0000 UTC m=+6.233615863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2b8b" (UniqueName: "kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b") pod "network-check-target-rsvsz" (UID: "47edcc66-8e41-41b6-9bb3-c457deebbda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:34.937773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:34.937746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:34.938224 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:34.937884 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:35.933834 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:35.933429 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:35.933834 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:35.933476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:35.933834 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:35.933574 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:35.933834 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:35.933682 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:36.378143 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:36.378061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:36.378143 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:36.378123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:36.378743 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.378261 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:36.378743 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.378273 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:36.378743 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.378319 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret podName:479c1d63-717e-42f7-b325-e3585799cd58 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:40.378300763 +0000 UTC m=+10.051498234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret") pod "global-pull-secret-syncer-9wmwb" (UID: "479c1d63-717e-42f7-b325-e3585799cd58") : object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:36.378743 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.378336 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs podName:6bd1a463-eb1a-4afb-aba9-ea8294291672 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:40.378327357 +0000 UTC m=+10.051524818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs") pod "network-metrics-daemon-jt82f" (UID: "6bd1a463-eb1a-4afb-aba9-ea8294291672") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:36.579908 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:36.579873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:36.580047 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.580021 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 17:00:36.580047 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.580040 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 17:00:36.580119 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.580053 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2b8b for pod openshift-network-diagnostics/network-check-target-rsvsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:36.580119 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.580105 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b podName:47edcc66-8e41-41b6-9bb3-c457deebbda9 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:40.580087585 +0000 UTC m=+10.253285044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2b8b" (UniqueName: "kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b") pod "network-check-target-rsvsz" (UID: "47edcc66-8e41-41b6-9bb3-c457deebbda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:36.935402 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:36.935256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:36.935599 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:36.935415 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:37.933255 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:37.933222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:37.933785 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:37.933340 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:37.933785 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:37.933689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:37.933884 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:37.933787 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:38.934274 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:38.934242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:38.934729 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:38.934401 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:39.933927 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:39.933847 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:39.934070 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:39.933974 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:39.934525 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:39.934507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:39.934833 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:39.934604 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:40.413177 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:40.412387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:40.413177 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:40.412454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:40.413177 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.412600 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:40.413177 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.412664 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret podName:479c1d63-717e-42f7-b325-e3585799cd58 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:48.412646148 +0000 UTC m=+18.085843607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret") pod "global-pull-secret-syncer-9wmwb" (UID: "479c1d63-717e-42f7-b325-e3585799cd58") : object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:40.413177 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.413033 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:40.413177 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.413077 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs podName:6bd1a463-eb1a-4afb-aba9-ea8294291672 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:48.413062695 +0000 UTC m=+18.086260154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs") pod "network-metrics-daemon-jt82f" (UID: "6bd1a463-eb1a-4afb-aba9-ea8294291672") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:40.614829 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:40.614351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:40.614829 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.614515 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 17:00:40.614829 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.614535 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 17:00:40.614829 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.614544 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2b8b for pod openshift-network-diagnostics/network-check-target-rsvsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:40.614829 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.614587 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b podName:47edcc66-8e41-41b6-9bb3-c457deebbda9 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:48.614573556 +0000 UTC m=+18.287771011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2b8b" (UniqueName: "kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b") pod "network-check-target-rsvsz" (UID: "47edcc66-8e41-41b6-9bb3-c457deebbda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:40.934273 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:40.933921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:40.934273 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:40.934038 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:41.933113 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:41.932999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:41.933113 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:41.933000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:41.933699 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:41.933141 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:41.933699 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:41.933198 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:42.933693 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:42.933664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:42.934057 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:42.933810 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:43.933499 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:43.933472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:43.933641 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:43.933472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:43.933641 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:43.933577 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:43.933734 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:43.933639 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:44.932983 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:44.932960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:44.933157 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:44.933088 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:45.933637 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:45.933609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:45.934078 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:45.933650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:45.934078 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:45.933722 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:45.934078 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:45.933873 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:46.933359 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:46.933327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:46.933548 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:46.933475 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:47.933111 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:47.933030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:47.933583 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:47.933031 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:47.933583 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:47.933149 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:47.933583 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:47.933253 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:48.474507 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:48.474473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:48.474777 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:48.474532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:48.474777 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.474661 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:48.474777 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.474734 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret podName:479c1d63-717e-42f7-b325-e3585799cd58 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:04.474712351 +0000 UTC m=+34.147909808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret") pod "global-pull-secret-syncer-9wmwb" (UID: "479c1d63-717e-42f7-b325-e3585799cd58") : object "kube-system"/"original-pull-secret" not registered Jul 30 17:00:48.474777 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.474750 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:48.475005 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.474800 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs podName:6bd1a463-eb1a-4afb-aba9-ea8294291672 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:04.474784561 +0000 UTC m=+34.147982021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs") pod "network-metrics-daemon-jt82f" (UID: "6bd1a463-eb1a-4afb-aba9-ea8294291672") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:00:48.676342 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:48.676305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:48.676530 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.676483 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 17:00:48.676530 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.676505 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 17:00:48.676530 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.676518 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2b8b for pod openshift-network-diagnostics/network-check-target-rsvsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:48.676697 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.676579 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b podName:47edcc66-8e41-41b6-9bb3-c457deebbda9 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:04.676561118 +0000 UTC m=+34.349758580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2b8b" (UniqueName: "kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b") pod "network-check-target-rsvsz" (UID: "47edcc66-8e41-41b6-9bb3-c457deebbda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:00:48.933091 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:48.933012 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:48.933256 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:48.933172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:49.933427 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:49.933386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:49.933882 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:49.933439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:49.933882 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:49.933510 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:49.933882 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:49.933653 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:50.934686 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:50.934649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:50.935133 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:50.934780 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:51.934863 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:51.934650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:51.935320 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:51.934751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:51.935320 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:51.935256 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:51.935469 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:51.935343 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:52.080073 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.080041 2576 generic.go:356] "Generic (PLEG): container finished" podID="db4dcb51-1f61-41d0-9f81-d5b71f4e1c49" containerID="ff499f612310b758cc08e2ddff9ffd180c0146da01baea9a7a359cacca6cd2e9" exitCode=0 Jul 30 17:00:52.080169 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.080112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerDied","Data":"ff499f612310b758cc08e2ddff9ffd180c0146da01baea9a7a359cacca6cd2e9"} Jul 30 17:00:52.081690 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.081652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qxth4" event={"ID":"063d41b9-4280-4bfa-9eed-4ce6ef4084d3","Type":"ContainerStarted","Data":"e82174d862853d6c93267175dd1f33d242d4612cb0a29c2fd0841cc240790101"} Jul 30 17:00:52.083175 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.083156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" event={"ID":"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18","Type":"ContainerStarted","Data":"dfbaf76af9363b5fd7eed6d7c4b04662418c67c770a10549775f4090fc28ea8a"} Jul 30 17:00:52.084481 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.084453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9hzs9" event={"ID":"8eb802f4-7b82-453c-8997-d9ff1248192c","Type":"ContainerStarted","Data":"7d630fb805cc3bdc0560c11d9ae10bdd106f65739a6191ebff244178b738391c"} Jul 30 17:00:52.087098 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.086925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"f4273b79576de2ad9358fe0388f4cecf34f88e8cd7116d751505ccb7561316f8"} Jul 30 17:00:52.087098 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.086994 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"352c6de98ba4108c9283edece836f21b6dbdc5cf87639ecac03160f61ae28cdf"} Jul 30 17:00:52.087098 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.087038 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"9bb612d6bcea4afc8b225fa625900baa9cd11b73c04888181bb47b2c2eba3ed5"} Jul 30 17:00:52.089522 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.089500 2576 generic.go:356] "Generic (PLEG): container finished" podID="ca7232da2ceebd3c2cd75f2c8e9f8e8f" containerID="b4c4b52ad1520d1e086abd585553ee18c7027410d588d88f0ae143cb7ffe8eba" exitCode=0 Jul 30 17:00:52.089612 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.089585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" event={"ID":"ca7232da2ceebd3c2cd75f2c8e9f8e8f","Type":"ContainerDied","Data":"b4c4b52ad1520d1e086abd585553ee18c7027410d588d88f0ae143cb7ffe8eba"} Jul 30 17:00:52.090884 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.090862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" event={"ID":"55e30e52bbfc0cb8ac949e33c9db102a","Type":"ContainerStarted","Data":"0fbeedf08876746e25574a62b8be9bfa7c1dd8133c8da97e2d48e33bb048b20c"} Jul 30 17:00:52.092204 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.092187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfrvd" event={"ID":"caf62b48-154a-4b0c-941b-506218889e2d","Type":"ContainerStarted","Data":"8376d813673397909e265919172cf7bd084954f46ce081a3f121e1a7cda09b0f"} Jul 30 17:00:52.093509 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.093487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v4kvx" event={"ID":"78e6945a-53cb-4b71-af2e-b91cdfdbcb90","Type":"ContainerStarted","Data":"6d82cb1548ddb1a4df18cfa8b016e60e508839e311c785068841b05c9f5b580c"} Jul 30 17:00:52.094797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.094780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" event={"ID":"9c3c11f0-4a5b-4291-9b07-daaae470ee87","Type":"ContainerStarted","Data":"070ba8165e0b6f35c6c70c9721e6254a83e707f562e2c27e7e0e8ac0037dd5c3"} Jul 30 17:00:52.124630 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.124595 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ghlw7" podStartSLOduration=2.610778066 podStartE2EDuration="21.124583373s" totalImagesPullingTime="18.513805307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:52.124473009 +0000 UTC m=+21.797670496" watchObservedRunningTime="2026-07-30 17:00:52.124583373 +0000 UTC m=+21.797780850" Jul 30 17:00:52.144526 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.144492 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qxth4" podStartSLOduration=2.630322462 podStartE2EDuration="21.144478859s" totalImagesPullingTime="18.514156397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:52.143923214 +0000 UTC m=+21.817120691" watchObservedRunningTime="2026-07-30 17:00:52.144478859 +0000 UTC m=+21.817676335" Jul 30 17:00:52.162784 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.162751 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rfrvd" podStartSLOduration=10.335011558 podStartE2EDuration="21.162740089s" totalImagesPullingTime="10.827728531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:52.162556311 +0000 UTC m=+21.835753789" watchObservedRunningTime="2026-07-30 17:00:52.162740089 +0000 UTC m=+21.835937566" Jul 30 17:00:52.178429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.178393 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9hzs9" podStartSLOduration=2.735453595 podStartE2EDuration="21.178366394s" totalImagesPullingTime="18.442912799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:52.177774534 +0000 UTC m=+21.850972002" watchObservedRunningTime="2026-07-30 17:00:52.178366394 +0000 UTC m=+21.851563871" Jul 30 17:00:52.195331 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.195299 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v4kvx" podStartSLOduration=2.6863229520000003 podStartE2EDuration="21.195288104s" totalImagesPullingTime="18.508965152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:52.19498387 +0000 UTC m=+21.868181346" watchObservedRunningTime="2026-07-30 17:00:52.195288104 +0000 UTC m=+21.868485584" Jul 30 17:00:52.299337 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.299301 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:00:52.378803 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.378768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:00:52.933551 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:52.933484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:52.933676 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:52.933604 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:53.100122 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.099907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"f720d164c861f04d737d050a889ce438c079081690dd7058b6066785bfa638b5"} Jul 30 17:00:53.100791 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.100175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"8c9219ddd35644c6b6670602b68717e78531f2aec63ae012888e45a4ac7e47ff"} Jul 30 17:00:53.100791 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.100223 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"8e17e4607444b08e1510c79a160408d781c5a6dff086b8ab2db18b95edc10cd2"} Jul 30 17:00:53.101272 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.101197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vv5k6" event={"ID":"d78dd639-1eea-42cb-9b55-b13ebcdbe59d","Type":"ContainerStarted","Data":"9730a75a5cbaf35655fb906c45fda7eb9dda047e12b262299071279770f8fceb"} Jul 30 17:00:53.117314 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.117270 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vv5k6" podStartSLOduration=11.300011053 podStartE2EDuration="22.117257954s" totalImagesPullingTime="10.817246901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:53.117112723 +0000 UTC m=+22.790310198" watchObservedRunningTime="2026-07-30 17:00:53.117257954 +0000 UTC m=+22.790455431" Jul 30 17:00:53.117597 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.117570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-254.ec2.internal" podStartSLOduration=22.117561316 podStartE2EDuration="22.117561316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:52.21244447 +0000 UTC m=+21.885641947" watchObservedRunningTime="2026-07-30 17:00:53.117561316 +0000 UTC m=+22.790759037" Jul 30 17:00:53.128444 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.128423 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:53.358489 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.358464 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 17:00:53.527316 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.527197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:00:53.584163 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.584130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:00:53.802585 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.802397 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T17:00:53.35848523Z","UUID":"10fdf444-b205-4613-9de3-939e62d06fa6","Handler":null,"Name":"","Endpoint":""} Jul 30 17:00:53.804246 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.804220 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 17:00:53.804385 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.804268 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 17:00:53.932991 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.932953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:53.933202 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:53.933089 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:53.933202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:53.933141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:53.933301 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:53.933256 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:54.105254 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:54.105174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" event={"ID":"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18","Type":"ContainerStarted","Data":"cf57b98ba5bb96e975a32a728f9cee76b9dcf8c46a87bca332d1741d2d7ec09a"} Jul 30 17:00:54.106966 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:54.106934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" event={"ID":"ca7232da2ceebd3c2cd75f2c8e9f8e8f","Type":"ContainerStarted","Data":"fc7399a128c5af94a71234bdf73f42ed9fab8e2abe338f964f599b156adeb41d"} Jul 30 17:00:54.933946 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:54.933908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:54.934136 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:54.934063 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:55.111180 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.111090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" event={"ID":"c94e5c6c-b5d3-4bb3-b52a-a6cd7dd3ed18","Type":"ContainerStarted","Data":"ecd087b8d6f8ee3b9aaa7193e60ae3183b9052eb169eec7d3e640f97fe322cbe"} Jul 30 17:00:55.114254 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.114231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"494e9255026758c6907500a369911da32de40ff983dc41e924ffd0803d1b9de7"} Jul 30 17:00:55.130805 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.130754 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-45gfz" podStartSLOduration=3.048723068 podStartE2EDuration="24.130741196s" totalImagesPullingTime="21.082018128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:55.130702803 +0000 UTC m=+24.803900279" watchObservedRunningTime="2026-07-30 17:00:55.130741196 +0000 UTC m=+24.803938672" Jul 30 17:00:55.131093 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.131069 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-254.ec2.internal" podStartSLOduration=24.131064809 podStartE2EDuration="24.131064809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:54.124417192 +0000 UTC m=+23.797614661" watchObservedRunningTime="2026-07-30 17:00:55.131064809 +0000 UTC m=+24.804262285" Jul 30 17:00:55.413987 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.413903 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:55.414624 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.414600 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:55.933078 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.933045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:55.933257 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:55.933059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:55.933257 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:55.933161 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:55.933364 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:55.933250 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:56.117672 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:56.117643 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9hzs9" Jul 30 17:00:56.181329 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:56.181297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:00:56.936225 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:56.936030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:56.936402 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:56.936320 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:57.120702 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.120614 2576 generic.go:356] "Generic (PLEG): container finished" podID="db4dcb51-1f61-41d0-9f81-d5b71f4e1c49" containerID="ce13aaa8bbfd70c9a87d278e392b97c4745bdd4a6b6cdc69956e4bdb3cb8f1e2" exitCode=0 Jul 30 17:00:57.121441 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.120699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerDied","Data":"ce13aaa8bbfd70c9a87d278e392b97c4745bdd4a6b6cdc69956e4bdb3cb8f1e2"} Jul 30 17:00:57.124790 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.124768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" event={"ID":"85ef8995-7981-46e0-b9b0-b0b7b4234643","Type":"ContainerStarted","Data":"5920a7b6736c550c946d232b8ca0586051634948c97ca67200c8f70069f52d62"} Jul 30 17:00:57.125039 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.125018 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:57.125588 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.125562 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:57.142573 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.142547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:57.142727 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.142660 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:57.175200 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.175149 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" podStartSLOduration=7.460105309 podStartE2EDuration="26.175130624s" totalImagesPullingTime="18.715025315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:57.174575363 +0000 UTC m=+26.847772851" watchObservedRunningTime="2026-07-30 17:00:57.175130624 +0000 UTC m=+26.848328101" Jul 30 17:00:57.934142 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.933358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:57.934142 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:57.933491 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:57.934142 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:57.933910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:57.934142 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:57.934044 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:58.075326 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:58.075289 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:00:58.933124 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:58.933095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:58.933713 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:58.933210 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:00:58.979856 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:58.979757 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rsvsz"] Jul 30 17:00:58.980024 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:58.979890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:00:58.980024 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:58.979992 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:00:58.980413 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:58.980391 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9wmwb"] Jul 30 17:00:58.980480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:58.980471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:00:58.980628 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:58.980551 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:00:58.981912 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:58.981891 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jt82f"] Jul 30 17:00:59.131832 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:59.131798 2576 generic.go:356] "Generic (PLEG): container finished" podID="db4dcb51-1f61-41d0-9f81-d5b71f4e1c49" containerID="f65dfebd4842e637f11d3e673eb8f709f0c3fbb78545b6479de46585e1387b72" exitCode=0 Jul 30 17:00:59.131999 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:59.131859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerDied","Data":"f65dfebd4842e637f11d3e673eb8f709f0c3fbb78545b6479de46585e1387b72"} Jul 30 17:00:59.131999 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:00:59.131938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:00:59.132091 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:00:59.132069 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:01:00.706256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:00.706227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:00.806692 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:00.806662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:00.906327 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:00.906296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:00.934337 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:00.934315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:00.934452 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:00.934429 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:00.934510 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:00.934461 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:01:00.934510 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:00.934472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:00.934577 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:00.934536 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:01:00.934646 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:00.934625 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:01:01.139282 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:01.139211 2576 generic.go:356] "Generic (PLEG): container finished" podID="db4dcb51-1f61-41d0-9f81-d5b71f4e1c49" containerID="a67d46206d1706ed3330f15fffd69c9d440e4941bab7bb00978e09936a1aaaa3" exitCode=0 Jul 30 17:01:01.139282 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:01.139267 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerDied","Data":"a67d46206d1706ed3330f15fffd69c9d440e4941bab7bb00978e09936a1aaaa3"} Jul 30 17:01:01.717184 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:01.717140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:01.771759 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:01.771725 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:02.738811 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:02.738772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:02.803100 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:02.803072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:02.933691 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:02.933656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:02.933854 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:02.933656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:02.933854 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:02.933795 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:01:02.933854 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:02.933805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:02.934079 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:02.933894 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:01:02.934079 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:02.933974 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:01:04.493899 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:04.493866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:04.494295 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:04.493931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:04.494295 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.493999 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:01:04.494295 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.494027 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 17:01:04.494295 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.494081 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs podName:6bd1a463-eb1a-4afb-aba9-ea8294291672 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:36.49405887 +0000 UTC m=+66.167256339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs") pod "network-metrics-daemon-jt82f" (UID: "6bd1a463-eb1a-4afb-aba9-ea8294291672") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 17:01:04.494295 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.494100 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret podName:479c1d63-717e-42f7-b325-e3585799cd58 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:36.494090485 +0000 UTC m=+66.167287955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret") pod "global-pull-secret-syncer-9wmwb" (UID: "479c1d63-717e-42f7-b325-e3585799cd58") : object "kube-system"/"original-pull-secret" not registered Jul 30 17:01:04.695537 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:04.695489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:04.695708 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.695630 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 17:01:04.695708 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.695653 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 17:01:04.695708 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.695665 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2b8b for pod openshift-network-diagnostics/network-check-target-rsvsz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:01:04.695829 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.695715 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b podName:47edcc66-8e41-41b6-9bb3-c457deebbda9 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:36.695701714 +0000 UTC m=+66.368899192 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2b8b" (UniqueName: "kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b") pod "network-check-target-rsvsz" (UID: "47edcc66-8e41-41b6-9bb3-c457deebbda9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 17:01:04.711072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:04.711041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:04.933198 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:04.933108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:04.933198 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:04.933124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:04.933447 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:04.933231 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:04.933447 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.933323 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jt82f" podUID="6bd1a463-eb1a-4afb-aba9-ea8294291672" Jul 30 17:01:04.933624 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.933591 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9wmwb" podUID="479c1d63-717e-42f7-b325-e3585799cd58" Jul 30 17:01:04.933732 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:04.933658 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsvsz" podUID="47edcc66-8e41-41b6-9bb3-c457deebbda9" Jul 30 17:01:05.625483 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.625282 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-254.ec2.internal" event="NodeReady" Jul 30 17:01:05.625847 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.625576 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 17:01:05.681246 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.681213 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2twfz"] Jul 30 17:01:05.682991 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.682976 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-446v7"] Jul 30 17:01:05.683112 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.683096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.684572 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.684551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:05.685987 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.685969 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bcbk6\"" Jul 30 17:01:05.686059 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.686004 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 17:01:05.686219 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.686204 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 17:01:05.687307 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.687253 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 17:01:05.687307 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.687279 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 17:01:05.687478 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.687281 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8ckwn\"" Jul 30 17:01:05.687478 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.687283 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 17:01:05.693326 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.693305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2twfz"] Jul 30 17:01:05.707508 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.707482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-446v7"] Jul 30 17:01:05.803340 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.803306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-config-volume\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.803520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.803349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e2f7dc-a057-4466-8c27-b93d291eff81-cert\") pod \"ingress-canary-446v7\" (UID: \"d1e2f7dc-a057-4466-8c27-b93d291eff81\") " pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:05.803520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.803398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmrh\" (UniqueName: \"kubernetes.io/projected/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-kube-api-access-4cmrh\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.803520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.803439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-tmp-dir\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.803520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.803461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.803520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.803512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjwxs\" (UniqueName: \"kubernetes.io/projected/d1e2f7dc-a057-4466-8c27-b93d291eff81-kube-api-access-kjwxs\") pod \"ingress-canary-446v7\" (UID: \"d1e2f7dc-a057-4466-8c27-b93d291eff81\") " pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:05.822187 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.822161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:05.885514 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.885430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:05.904338 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.904317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-config-volume\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.904445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.904350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e2f7dc-a057-4466-8c27-b93d291eff81-cert\") pod \"ingress-canary-446v7\" (UID: \"d1e2f7dc-a057-4466-8c27-b93d291eff81\") " pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:05.904445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.904399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmrh\" (UniqueName: \"kubernetes.io/projected/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-kube-api-access-4cmrh\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.904445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.904432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-tmp-dir\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.904611 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.904447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.904611 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.904466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwxs\" (UniqueName: \"kubernetes.io/projected/d1e2f7dc-a057-4466-8c27-b93d291eff81-kube-api-access-kjwxs\") pod \"ingress-canary-446v7\" (UID: \"d1e2f7dc-a057-4466-8c27-b93d291eff81\") " pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:05.904611 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:05.904545 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 17:01:05.904611 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:05.904608 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls podName:643e3b9d-d9a0-401d-bc08-0ba5554bbba8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:06.404588792 +0000 UTC m=+36.077786249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls") pod "dns-default-2twfz" (UID: "643e3b9d-d9a0-401d-bc08-0ba5554bbba8") : secret "dns-default-metrics-tls" not found Jul 30 17:01:05.904997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.904965 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-tmp-dir\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.905207 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.905186 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-config-volume\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.908885 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.908869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e2f7dc-a057-4466-8c27-b93d291eff81-cert\") pod \"ingress-canary-446v7\" (UID: \"d1e2f7dc-a057-4466-8c27-b93d291eff81\") " pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:05.913690 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.913671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwxs\" (UniqueName: \"kubernetes.io/projected/d1e2f7dc-a057-4466-8c27-b93d291eff81-kube-api-access-kjwxs\") pod \"ingress-canary-446v7\" (UID: \"d1e2f7dc-a057-4466-8c27-b93d291eff81\") " pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:05.913779 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.913716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmrh\" (UniqueName: \"kubernetes.io/projected/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-kube-api-access-4cmrh\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:05.998916 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:05.998864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-446v7" Jul 30 17:01:06.130677 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.130644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-446v7"] Jul 30 17:01:06.133951 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:06.133918 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e2f7dc_a057_4466_8c27_b93d291eff81.slice/crio-1c0d0ce09f849e203a123d369893705b758524035eeaaa0fcc04591ad48d96dd WatchSource:0}: Error finding container 1c0d0ce09f849e203a123d369893705b758524035eeaaa0fcc04591ad48d96dd: Status 404 returned error can't find the container with id 1c0d0ce09f849e203a123d369893705b758524035eeaaa0fcc04591ad48d96dd Jul 30 17:01:06.154207 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.154180 2576 generic.go:356] "Generic (PLEG): container finished" podID="db4dcb51-1f61-41d0-9f81-d5b71f4e1c49" containerID="a54a33a8e9e36fffa16758e15a6f41fc69e8468cf5a7f9d3f322d5cbfc4887ee" exitCode=0 Jul 30 17:01:06.154293 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.154249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerDied","Data":"a54a33a8e9e36fffa16758e15a6f41fc69e8468cf5a7f9d3f322d5cbfc4887ee"} Jul 30 17:01:06.155415 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.155390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-446v7" event={"ID":"d1e2f7dc-a057-4466-8c27-b93d291eff81","Type":"ContainerStarted","Data":"1c0d0ce09f849e203a123d369893705b758524035eeaaa0fcc04591ad48d96dd"} Jul 30 17:01:06.409078 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.408977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:06.409224 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:06.409130 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 17:01:06.409224 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:06.409195 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls podName:643e3b9d-d9a0-401d-bc08-0ba5554bbba8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:07.40917873 +0000 UTC m=+37.082376185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls") pod "dns-default-2twfz" (UID: "643e3b9d-d9a0-401d-bc08-0ba5554bbba8") : secret "dns-default-metrics-tls" not found Jul 30 17:01:06.936930 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.936894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:06.936930 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.936929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:06.937757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.936894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:06.940961 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.940934 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 17:01:06.941096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.940975 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcvkj\"" Jul 30 17:01:06.941096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.940998 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 17:01:06.941096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.941061 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 17:01:06.941297 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.941279 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 17:01:06.941383 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:06.941281 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dnsfv\"" Jul 30 17:01:07.161902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:07.161643 2576 generic.go:356] "Generic (PLEG): container finished" podID="db4dcb51-1f61-41d0-9f81-d5b71f4e1c49" containerID="b98fff57f0036d9972dd59f203c56c77f7652328c7e855c21ce193c64463f5c3" exitCode=0 Jul 30 17:01:07.161902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:07.161726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerDied","Data":"b98fff57f0036d9972dd59f203c56c77f7652328c7e855c21ce193c64463f5c3"} Jul 30 17:01:07.417692 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:07.417656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:07.417867 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:07.417846 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 17:01:07.417953 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:07.417940 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls podName:643e3b9d-d9a0-401d-bc08-0ba5554bbba8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:09.417917898 +0000 UTC m=+39.091115355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls") pod "dns-default-2twfz" (UID: "643e3b9d-d9a0-401d-bc08-0ba5554bbba8") : secret "dns-default-metrics-tls" not found Jul 30 17:01:08.167764 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:08.167726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzck8" event={"ID":"db4dcb51-1f61-41d0-9f81-d5b71f4e1c49","Type":"ContainerStarted","Data":"64b8673d6b37312f35f396e6d0288a9b88f4c91d593bc25f477d8031e8b1f596"} Jul 30 17:01:08.196236 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:08.196176 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lzck8" podStartSLOduration=7.28341325 podStartE2EDuration="37.196156048s" totalImagesPullingTime="29.912742798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:08.195202337 +0000 UTC m=+37.868399814" watchObservedRunningTime="2026-07-30 17:01:08.196156048 +0000 UTC m=+37.869353525" Jul 30 17:01:08.319061 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:08.319031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:08.378768 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:08.378726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:09.171531 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:09.171438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-446v7" event={"ID":"d1e2f7dc-a057-4466-8c27-b93d291eff81","Type":"ContainerStarted","Data":"c6857dd6e51361a6c7387704a4e9339c349076d34ce9c8e594cbb9be47d189db"} Jul 30 17:01:09.189821 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:09.189771 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-446v7" podStartSLOduration=1.833012209 podStartE2EDuration="4.18975731s" totalImagesPullingTime="2.356745101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:09.188591315 +0000 UTC m=+38.861788791" watchObservedRunningTime="2026-07-30 17:01:09.18975731 +0000 UTC m=+38.862954781" Jul 30 17:01:09.285251 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:09.285213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:09.433635 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:09.433547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:09.433786 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:09.433654 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 17:01:09.433786 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:09.433714 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls podName:643e3b9d-d9a0-401d-bc08-0ba5554bbba8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:13.433699483 +0000 UTC m=+43.106896939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls") pod "dns-default-2twfz" (UID: "643e3b9d-d9a0-401d-bc08-0ba5554bbba8") : secret "dns-default-metrics-tls" not found Jul 30 17:01:12.680139 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:12.680105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:12.760941 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:12.760908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:13.457357 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:13.457315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:13.459230 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:13.459212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:13.459321 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:13.459309 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 17:01:13.459383 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:13.459362 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls podName:643e3b9d-d9a0-401d-bc08-0ba5554bbba8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:21.459348713 +0000 UTC m=+51.132546168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls") pod "dns-default-2twfz" (UID: "643e3b9d-d9a0-401d-bc08-0ba5554bbba8") : secret "dns-default-metrics-tls" not found Jul 30 17:01:13.557994 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:13.557969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:15.033575 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.033544 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-c795x"] Jul 30 17:01:15.064694 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.064667 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-c795x"] Jul 30 17:01:15.064819 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.064755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.067505 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.067477 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 30 17:01:15.067505 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.067490 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 30 17:01:15.067690 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.067530 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 17:01:15.067690 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.067595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 17:01:15.068738 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.068722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hd5gx\"" Jul 30 17:01:15.070244 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.070219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec7ff92-af94-4d0f-ace3-5d7adef84338-tmp\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.070339 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.070263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec7ff92-af94-4d0f-ace3-5d7adef84338-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.070339 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.070310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ec7ff92-af94-4d0f-ace3-5d7adef84338-snapshots\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.070481 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.070345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec7ff92-af94-4d0f-ace3-5d7adef84338-serving-cert\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.070481 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.070369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec7ff92-af94-4d0f-ace3-5d7adef84338-service-ca-bundle\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.070481 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.070453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdtk\" (UniqueName: \"kubernetes.io/projected/3ec7ff92-af94-4d0f-ace3-5d7adef84338-kube-api-access-nfdtk\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.073938 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.073920 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 30 17:01:15.132454 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.132432 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h"] Jul 30 17:01:15.157435 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.157413 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp"] Jul 30 17:01:15.157556 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.157539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" Jul 30 17:01:15.160558 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.160541 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 30 17:01:15.160661 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.160630 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dl75s\"" Jul 30 17:01:15.160726 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.160666 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 30 17:01:15.171049 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec7ff92-af94-4d0f-ace3-5d7adef84338-tmp\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171146 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frkdw\" (UniqueName: \"kubernetes.io/projected/07b13ac7-3780-4f15-892e-49fff37fec07-kube-api-access-frkdw\") pod \"volume-data-source-validator-549c6d54cf-mrf7h\" (UID: \"07b13ac7-3780-4f15-892e-49fff37fec07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" Jul 30 17:01:15.171146 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec7ff92-af94-4d0f-ace3-5d7adef84338-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171146 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ec7ff92-af94-4d0f-ace3-5d7adef84338-snapshots\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171307 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec7ff92-af94-4d0f-ace3-5d7adef84338-serving-cert\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171307 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec7ff92-af94-4d0f-ace3-5d7adef84338-service-ca-bundle\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171307 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdtk\" (UniqueName: \"kubernetes.io/projected/3ec7ff92-af94-4d0f-ace3-5d7adef84338-kube-api-access-nfdtk\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171486 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec7ff92-af94-4d0f-ace3-5d7adef84338-tmp\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171661 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171643 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ec7ff92-af94-4d0f-ace3-5d7adef84338-snapshots\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.171985 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.171962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec7ff92-af94-4d0f-ace3-5d7adef84338-service-ca-bundle\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.172479 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.172459 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec7ff92-af94-4d0f-ace3-5d7adef84338-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.174770 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.174742 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec7ff92-af94-4d0f-ace3-5d7adef84338-serving-cert\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.179768 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.179742 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m"] Jul 30 17:01:15.181925 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.181870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" Jul 30 17:01:15.184741 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.184723 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wvg8p\"" Jul 30 17:01:15.184741 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.184731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdtk\" (UniqueName: \"kubernetes.io/projected/3ec7ff92-af94-4d0f-ace3-5d7adef84338-kube-api-access-nfdtk\") pod \"insights-operator-5885fdb759-c795x\" (UID: \"3ec7ff92-af94-4d0f-ace3-5d7adef84338\") " pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.192429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.192401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:15.204273 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.204252 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk"] Jul 30 17:01:15.204417 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.204397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:15.206784 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.206764 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rp9dr\"" Jul 30 17:01:15.206784 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.206775 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 30 17:01:15.206919 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.206765 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 30 17:01:15.206919 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.206765 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 30 17:01:15.227942 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.227924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.227942 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.227926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h"] Jul 30 17:01:15.228053 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.228006 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp"] Jul 30 17:01:15.228092 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.228057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk"] Jul 30 17:01:15.228092 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.228084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m"] Jul 30 17:01:15.230750 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.230730 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 30 17:01:15.231069 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.231050 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 30 17:01:15.231069 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.231059 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 30 17:01:15.231192 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.231111 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8sbzm\"" Jul 30 17:01:15.231192 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.231112 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 30 17:01:15.231732 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.231718 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl"] Jul 30 17:01:15.252072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.252052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl"] Jul 30 17:01:15.252174 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.252140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.254478 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.254456 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 30 17:01:15.254590 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.254456 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 30 17:01:15.254652 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.254457 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 30 17:01:15.254652 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.254604 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d5vzd\"" Jul 30 17:01:15.254775 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.254757 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 30 17:01:15.255102 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.255080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:15.272290 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pznxv\" (UniqueName: \"kubernetes.io/projected/8222de4a-dfe0-41eb-8857-142fa0f5919a-kube-api-access-pznxv\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:15.272402 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.272402 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frkdw\" (UniqueName: \"kubernetes.io/projected/07b13ac7-3780-4f15-892e-49fff37fec07-kube-api-access-frkdw\") pod \"volume-data-source-validator-549c6d54cf-mrf7h\" (UID: \"07b13ac7-3780-4f15-892e-49fff37fec07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" Jul 30 17:01:15.272402 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2633ff8f-337d-48db-ace0-2316e62d08b5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.272517 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2633ff8f-337d-48db-ace0-2316e62d08b5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.272517 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtr5\" (UniqueName: \"kubernetes.io/projected/dbb3afa1-388f-48d4-a3fe-3565261c075f-kube-api-access-nvtr5\") pod \"network-check-source-7d88cb55d8-8bkcp\" (UID: \"dbb3afa1-388f-48d4-a3fe-3565261c075f\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" Jul 30 17:01:15.272517 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szm22\" (UniqueName: \"kubernetes.io/projected/2633ff8f-337d-48db-ace0-2316e62d08b5-kube-api-access-szm22\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.272517 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-config\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.272517 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:15.272659 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.272575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4gsq\" (UniqueName: \"kubernetes.io/projected/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-kube-api-access-l4gsq\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.280732 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.280714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frkdw\" (UniqueName: \"kubernetes.io/projected/07b13ac7-3780-4f15-892e-49fff37fec07-kube-api-access-frkdw\") pod \"volume-data-source-validator-549c6d54cf-mrf7h\" (UID: \"07b13ac7-3780-4f15-892e-49fff37fec07\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" Jul 30 17:01:15.373285 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pznxv\" (UniqueName: \"kubernetes.io/projected/8222de4a-dfe0-41eb-8857-142fa0f5919a-kube-api-access-pznxv\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:15.373285 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.373285 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2633ff8f-337d-48db-ace0-2316e62d08b5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.373510 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2633ff8f-337d-48db-ace0-2316e62d08b5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.373510 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtr5\" (UniqueName: \"kubernetes.io/projected/dbb3afa1-388f-48d4-a3fe-3565261c075f-kube-api-access-nvtr5\") pod \"network-check-source-7d88cb55d8-8bkcp\" (UID: \"dbb3afa1-388f-48d4-a3fe-3565261c075f\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" Jul 30 17:01:15.373510 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szm22\" (UniqueName: \"kubernetes.io/projected/2633ff8f-337d-48db-ace0-2316e62d08b5-kube-api-access-szm22\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.373510 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-c795x" Jul 30 17:01:15.373704 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-config\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.373757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:15.373816 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.373799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4gsq\" (UniqueName: \"kubernetes.io/projected/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-kube-api-access-l4gsq\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.373881 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:15.373862 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 17:01:15.373956 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:15.373943 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls podName:8222de4a-dfe0-41eb-8857-142fa0f5919a nodeName:}" failed. No retries permitted until 2026-07-30 17:01:15.873925339 +0000 UTC m=+45.547122818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls") pod "cluster-samples-operator-6748489656-p472m" (UID: "8222de4a-dfe0-41eb-8857-142fa0f5919a") : secret "samples-operator-tls" not found Jul 30 17:01:15.374030 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.374011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2633ff8f-337d-48db-ace0-2316e62d08b5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.374090 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.374073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-config\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.375463 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.375445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-serving-cert\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.375532 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.375523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2633ff8f-337d-48db-ace0-2316e62d08b5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.383970 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.383950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtr5\" (UniqueName: \"kubernetes.io/projected/dbb3afa1-388f-48d4-a3fe-3565261c075f-kube-api-access-nvtr5\") pod \"network-check-source-7d88cb55d8-8bkcp\" (UID: \"dbb3afa1-388f-48d4-a3fe-3565261c075f\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" Jul 30 17:01:15.384058 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.384032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szm22\" (UniqueName: \"kubernetes.io/projected/2633ff8f-337d-48db-ace0-2316e62d08b5-kube-api-access-szm22\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl\" (UID: \"2633ff8f-337d-48db-ace0-2316e62d08b5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.384058 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.384037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4gsq\" (UniqueName: \"kubernetes.io/projected/45bb5507-77d6-4be7-bc89-95aa2d5f02b6-kube-api-access-l4gsq\") pod \"service-ca-operator-5d6c75f84f-p2jdk\" (UID: \"45bb5507-77d6-4be7-bc89-95aa2d5f02b6\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.384290 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.384273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pznxv\" (UniqueName: \"kubernetes.io/projected/8222de4a-dfe0-41eb-8857-142fa0f5919a-kube-api-access-pznxv\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:15.466033 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.465990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" Jul 30 17:01:15.494733 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.494700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" Jul 30 17:01:15.523521 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.523342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-c795x"] Jul 30 17:01:15.535832 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.535808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" Jul 30 17:01:15.538444 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:15.538418 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec7ff92_af94_4d0f_ace3_5d7adef84338.slice/crio-1dc6827973cdf08068c31c0012bcf64831ac55b361b7eb715a71d05cf766796b WatchSource:0}: Error finding container 1dc6827973cdf08068c31c0012bcf64831ac55b361b7eb715a71d05cf766796b: Status 404 returned error can't find the container with id 1dc6827973cdf08068c31c0012bcf64831ac55b361b7eb715a71d05cf766796b Jul 30 17:01:15.560878 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.560851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" Jul 30 17:01:15.633646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.633498 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h"] Jul 30 17:01:15.692698 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.692666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp"] Jul 30 17:01:15.695704 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:15.695672 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb3afa1_388f_48d4_a3fe_3565261c075f.slice/crio-01067c8088fbddc976479edf282a978ffcbc449810f24c91aab0d0d68d2b0633 WatchSource:0}: Error finding container 01067c8088fbddc976479edf282a978ffcbc449810f24c91aab0d0d68d2b0633: Status 404 returned error can't find the container with id 01067c8088fbddc976479edf282a978ffcbc449810f24c91aab0d0d68d2b0633 Jul 30 17:01:15.732583 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.732563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk"] Jul 30 17:01:15.734773 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:15.734744 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bb5507_77d6_4be7_bc89_95aa2d5f02b6.slice/crio-9e05eaea564bc9f99e4e536e920ff99b86c479f32ba7b81378a273529010ca87 WatchSource:0}: Error finding container 9e05eaea564bc9f99e4e536e920ff99b86c479f32ba7b81378a273529010ca87: Status 404 returned error can't find the container with id 9e05eaea564bc9f99e4e536e920ff99b86c479f32ba7b81378a273529010ca87 Jul 30 17:01:15.753298 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.753280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl"] Jul 30 17:01:15.755988 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:15.755968 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2633ff8f_337d_48db_ace0_2316e62d08b5.slice/crio-d36cd9122af7388beb24070ec6e4a19fe2c050732d5ef35bf039c530e041951b WatchSource:0}: Error finding container d36cd9122af7388beb24070ec6e4a19fe2c050732d5ef35bf039c530e041951b: Status 404 returned error can't find the container with id d36cd9122af7388beb24070ec6e4a19fe2c050732d5ef35bf039c530e041951b Jul 30 17:01:15.878465 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:15.878438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:15.878573 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:15.878548 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 17:01:15.878619 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:15.878595 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls podName:8222de4a-dfe0-41eb-8857-142fa0f5919a nodeName:}" failed. No retries permitted until 2026-07-30 17:01:16.878581571 +0000 UTC m=+46.551779026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls") pod "cluster-samples-operator-6748489656-p472m" (UID: "8222de4a-dfe0-41eb-8857-142fa0f5919a") : secret "samples-operator-tls" not found Jul 30 17:01:16.188566 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:16.188526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" event={"ID":"45bb5507-77d6-4be7-bc89-95aa2d5f02b6","Type":"ContainerStarted","Data":"9e05eaea564bc9f99e4e536e920ff99b86c479f32ba7b81378a273529010ca87"} Jul 30 17:01:16.190560 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:16.190531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" event={"ID":"07b13ac7-3780-4f15-892e-49fff37fec07","Type":"ContainerStarted","Data":"9b5b0ecf1b708b76f5924084ac63ec960bd5d1943ef9dc5150b55ee9c5fb9344"} Jul 30 17:01:16.192077 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:16.192048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c795x" event={"ID":"3ec7ff92-af94-4d0f-ace3-5d7adef84338","Type":"ContainerStarted","Data":"1dc6827973cdf08068c31c0012bcf64831ac55b361b7eb715a71d05cf766796b"} Jul 30 17:01:16.193586 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:16.193541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" event={"ID":"2633ff8f-337d-48db-ace0-2316e62d08b5","Type":"ContainerStarted","Data":"d36cd9122af7388beb24070ec6e4a19fe2c050732d5ef35bf039c530e041951b"} Jul 30 17:01:16.194732 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:16.194703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" event={"ID":"dbb3afa1-388f-48d4-a3fe-3565261c075f","Type":"ContainerStarted","Data":"01067c8088fbddc976479edf282a978ffcbc449810f24c91aab0d0d68d2b0633"} Jul 30 17:01:16.886328 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:16.886291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:16.886535 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:16.886501 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 17:01:16.886606 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:16.886559 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls podName:8222de4a-dfe0-41eb-8857-142fa0f5919a nodeName:}" failed. No retries permitted until 2026-07-30 17:01:18.886541632 +0000 UTC m=+48.559739087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls") pod "cluster-samples-operator-6748489656-p472m" (UID: "8222de4a-dfe0-41eb-8857-142fa0f5919a") : secret "samples-operator-tls" not found Jul 30 17:01:17.380557 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:17.380248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:17.441318 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:17.441277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:18.903936 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:18.903897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:18.904318 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:18.904064 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 17:01:18.904318 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:18.904136 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls podName:8222de4a-dfe0-41eb-8857-142fa0f5919a nodeName:}" failed. No retries permitted until 2026-07-30 17:01:22.904115863 +0000 UTC m=+52.577313318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls") pod "cluster-samples-operator-6748489656-p472m" (UID: "8222de4a-dfe0-41eb-8857-142fa0f5919a") : secret "samples-operator-tls" not found Jul 30 17:01:20.844980 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:20.844611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:20.946395 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:20.946289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:21.217511 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.217155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" event={"ID":"45bb5507-77d6-4be7-bc89-95aa2d5f02b6","Type":"ContainerStarted","Data":"196c111707c53ea73ef8e0e95370e155dad0cf731c033baabac651c76fcf9967"} Jul 30 17:01:21.218828 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.218794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" event={"ID":"07b13ac7-3780-4f15-892e-49fff37fec07","Type":"ContainerStarted","Data":"b4516a6eda9fb6c723d462a0ff527b79262e37d5c05c4ec0a295e60c1d3acecb"} Jul 30 17:01:21.220650 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.220626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c795x" event={"ID":"3ec7ff92-af94-4d0f-ace3-5d7adef84338","Type":"ContainerStarted","Data":"e13f74dad5f4f324f3fdeb110ef4c2354ae7f1f2ec3751d3af399ec6d264c00c"} Jul 30 17:01:21.223112 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.223090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" event={"ID":"2633ff8f-337d-48db-ace0-2316e62d08b5","Type":"ContainerStarted","Data":"814881cb60b4fbbc62191af7ae882abe35658bea409e49afacd8a0872373abd9"} Jul 30 17:01:21.224610 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.224587 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" event={"ID":"dbb3afa1-388f-48d4-a3fe-3565261c075f","Type":"ContainerStarted","Data":"49a84c7da3eec4179ee703ef208b458e21e4035941836245d921a5d334d621ae"} Jul 30 17:01:21.235020 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.234978 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" podStartSLOduration=1.218502527 podStartE2EDuration="6.23496244s" totalImagesPullingTime="5.016459913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:21.233726381 +0000 UTC m=+50.906923859" watchObservedRunningTime="2026-07-30 17:01:21.23496244 +0000 UTC m=+50.908159919" Jul 30 17:01:21.254397 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.254337 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-c795x" podStartSLOduration=1.043208138 podStartE2EDuration="6.254322111s" totalImagesPullingTime="5.211113973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:21.253508127 +0000 UTC m=+50.926705608" watchObservedRunningTime="2026-07-30 17:01:21.254322111 +0000 UTC m=+50.927519589" Jul 30 17:01:21.289071 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.288402 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mrf7h" podStartSLOduration=1.2068523 podStartE2EDuration="6.288387255s" totalImagesPullingTime="5.081534955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:21.288012366 +0000 UTC m=+50.961209839" watchObservedRunningTime="2026-07-30 17:01:21.288387255 +0000 UTC m=+50.961584723" Jul 30 17:01:21.289322 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.289291 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" podStartSLOduration=1.288373293 podStartE2EDuration="6.289282243s" totalImagesPullingTime="5.00090895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:21.271857212 +0000 UTC m=+50.945054689" watchObservedRunningTime="2026-07-30 17:01:21.289282243 +0000 UTC m=+50.962479724" Jul 30 17:01:21.306407 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.306312 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8bkcp" podStartSLOduration=1.250748588 podStartE2EDuration="6.306298841s" totalImagesPullingTime="5.055550253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:21.305726173 +0000 UTC m=+50.978923647" watchObservedRunningTime="2026-07-30 17:01:21.306298841 +0000 UTC m=+50.979496322" Jul 30 17:01:21.525213 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.525133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:21.525369 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:21.525286 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 17:01:21.525369 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:21.525360 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls podName:643e3b9d-d9a0-401d-bc08-0ba5554bbba8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:37.525340961 +0000 UTC m=+67.198538442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls") pod "dns-default-2twfz" (UID: "643e3b9d-d9a0-401d-bc08-0ba5554bbba8") : secret "dns-default-metrics-tls" not found Jul 30 17:01:21.590283 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.590252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:21.690583 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.690548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:21.938157 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:21.938075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:22.175142 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:22.175112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:22.229264 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:22.229235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:22.779828 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:22.779796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:22.891197 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:22.891160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:22.935932 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:22.935890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:22.936130 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:22.936032 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 17:01:22.936207 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:22.936193 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls podName:8222de4a-dfe0-41eb-8857-142fa0f5919a nodeName:}" failed. No retries permitted until 2026-07-30 17:01:30.936174793 +0000 UTC m=+60.609372250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls") pod "cluster-samples-operator-6748489656-p472m" (UID: "8222de4a-dfe0-41eb-8857-142fa0f5919a") : secret "samples-operator-tls" not found Jul 30 17:01:22.945496 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:22.945460 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:23.001426 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.001392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:23.131596 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.131517 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2ppfb"] Jul 30 17:01:23.152618 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.152594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ppfb"] Jul 30 17:01:23.152796 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.152701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.155697 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.155676 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 17:01:23.155697 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.155683 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brp5b\"" Jul 30 17:01:23.155852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.155680 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 17:01:23.161810 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.161787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:23.224915 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.224881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:23.240599 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.240571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.240728 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.240602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-crio-socket\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.240728 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.240623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccxf\" (UniqueName: \"kubernetes.io/projected/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-api-access-gccxf\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.240728 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.240675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.240850 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.240739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-data-volume\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.342016 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.341981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-data-volume\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.342191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.342092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.342191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.342121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-crio-socket\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.342191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.342149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccxf\" (UniqueName: \"kubernetes.io/projected/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-api-access-gccxf\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.342191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.342177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.342392 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.342243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-crio-socket\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.342392 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.342263 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 17:01:23.342392 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.342291 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 17:01:23.342392 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.342340 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:23.842324465 +0000 UTC m=+53.515521920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : secret "insights-runtime-extractor-tls" not found Jul 30 17:01:23.342392 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.342361 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:23.842347208 +0000 UTC m=+53.515544663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : configmap "kube-rbac-proxy" not found Jul 30 17:01:23.342392 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.342356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-data-volume\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.350618 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.350600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccxf\" (UniqueName: \"kubernetes.io/projected/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-api-access-gccxf\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.846719 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.846687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.846973 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:23.846735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:23.846973 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.846845 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 17:01:23.846973 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.846870 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 17:01:23.846973 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.846897 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:24.84688374 +0000 UTC m=+54.520081195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : configmap "kube-rbac-proxy" not found Jul 30 17:01:23.846973 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:23.846946 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:24.846924166 +0000 UTC m=+54.520121635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : secret "insights-runtime-extractor-tls" not found Jul 30 17:01:24.243243 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.243211 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-s5vdw"] Jul 30 17:01:24.246191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.246173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.248944 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.248922 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 30 17:01:24.249583 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.249567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-sgbsg\"" Jul 30 17:01:24.249666 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.249585 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 30 17:01:24.249666 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.249646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 30 17:01:24.249774 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.249652 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 30 17:01:24.253919 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.253899 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-s5vdw"] Jul 30 17:01:24.286773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.286747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:24.338705 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.338684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:24.350827 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.350803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-signing-key\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.350938 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.350923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-signing-cabundle\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.350998 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.350982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb6zm\" (UniqueName: \"kubernetes.io/projected/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-kube-api-access-xb6zm\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.452102 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.452071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb6zm\" (UniqueName: \"kubernetes.io/projected/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-kube-api-access-xb6zm\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.452236 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.452124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-signing-key\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.452325 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.452304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-signing-cabundle\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.452915 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.452896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-signing-cabundle\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.454451 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.454432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-signing-key\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.461276 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.461256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb6zm\" (UniqueName: \"kubernetes.io/projected/a1dfde55-4e99-441b-a134-6bd71b6ab2f6-kube-api-access-xb6zm\") pod \"service-ca-658d7b7c78-s5vdw\" (UID: \"a1dfde55-4e99-441b-a134-6bd71b6ab2f6\") " pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.555269 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.555186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" Jul 30 17:01:24.677279 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.677248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-s5vdw"] Jul 30 17:01:24.680193 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:24.680163 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dfde55_4e99_441b_a134_6bd71b6ab2f6.slice/crio-4967c327e071d8ae9d049eeb8ba352b87aee61cc8ea88b5e8736367ae43a6e6f WatchSource:0}: Error finding container 4967c327e071d8ae9d049eeb8ba352b87aee61cc8ea88b5e8736367ae43a6e6f: Status 404 returned error can't find the container with id 4967c327e071d8ae9d049eeb8ba352b87aee61cc8ea88b5e8736367ae43a6e6f Jul 30 17:01:24.855238 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.855141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:24.855430 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:24.855253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:24.855430 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:24.855275 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 17:01:24.855430 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:24.855349 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 17:01:24.855430 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:24.855350 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:26.855335188 +0000 UTC m=+56.528532642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : configmap "kube-rbac-proxy" not found Jul 30 17:01:24.855430 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:24.855425 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:26.85540802 +0000 UTC m=+56.528605482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : secret "insights-runtime-extractor-tls" not found Jul 30 17:01:25.240995 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:25.240966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" event={"ID":"a1dfde55-4e99-441b-a134-6bd71b6ab2f6","Type":"ContainerStarted","Data":"b489377d58c8e24a5404a0d04181b1602ce7d851589250eb217bce6d5ff9938c"} Jul 30 17:01:25.241154 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:25.241013 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" event={"ID":"a1dfde55-4e99-441b-a134-6bd71b6ab2f6","Type":"ContainerStarted","Data":"4967c327e071d8ae9d049eeb8ba352b87aee61cc8ea88b5e8736367ae43a6e6f"} Jul 30 17:01:25.266037 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:25.265996 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-s5vdw" podStartSLOduration=1.2659822 podStartE2EDuration="1.2659822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:25.265794482 +0000 UTC m=+54.938991960" watchObservedRunningTime="2026-07-30 17:01:25.2659822 +0000 UTC m=+54.939179677" Jul 30 17:01:25.354000 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:25.353957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:25.965619 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:25.965586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:26.110851 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.110809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:26.113749 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.113724 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfrvd_caf62b48-154a-4b0c-941b-506218889e2d/dns-node-resolver/0.log" Jul 30 17:01:26.214359 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.214327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:26.374899 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.374814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:26.629993 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.629889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:26.874341 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.874292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:26.874341 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.874345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:26.874625 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:26.874435 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 17:01:26.874625 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:26.874479 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 17:01:26.874625 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:26.874488 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:30.874474785 +0000 UTC m=+60.547672239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : configmap "kube-rbac-proxy" not found Jul 30 17:01:26.874625 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:26.874564 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:30.874547272 +0000 UTC m=+60.547744730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : secret "insights-runtime-extractor-tls" not found Jul 30 17:01:26.910073 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:26.910006 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v4kvx_78e6945a-53cb-4b71-af2e-b91cdfdbcb90/node-ca/0.log" Jul 30 17:01:27.310168 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:27.310128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-446v7_d1e2f7dc-a057-4466-8c27-b93d291eff81/serve-healthcheck-canary/0.log" Jul 30 17:01:27.623354 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:27.623262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:28.017201 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:28.017148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:28.222212 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:28.222176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:28.312142 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:28.312050 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl_2633ff8f-337d-48db-ace0-2316e62d08b5/kube-storage-version-migrator-operator/0.log" Jul 30 17:01:28.422739 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:28.422697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:28.627449 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:28.627357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:28.683968 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:28.683936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:28.939600 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:28.939496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:29.151869 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:29.151844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p8znm" Jul 30 17:01:29.429819 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:29.429786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:29.618847 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:29.618811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:29.873705 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:29.873669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:29.935243 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:29.935213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:30.224283 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:30.224251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:30.428388 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:30.428342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:30.782088 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:30.782055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:30.906471 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:30.906365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:30.906882 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:30.906495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:30.906882 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:30.906503 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 17:01:30.906882 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:30.906558 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:38.906545021 +0000 UTC m=+68.579742476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : secret "insights-runtime-extractor-tls" not found Jul 30 17:01:30.906882 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:30.906612 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 17:01:30.906882 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:01:30.906679 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm podName:6eaca9fc-0810-4b49-9e7b-b887e6407fc1 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:38.906664052 +0000 UTC m=+68.579861513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ppfb" (UID: "6eaca9fc-0810-4b49-9e7b-b887e6407fc1") : configmap "kube-rbac-proxy" not found Jul 30 17:01:30.929867 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:30.929823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:31.008002 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:31.007965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:31.010257 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:31.010234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8222de4a-dfe0-41eb-8857-142fa0f5919a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p472m\" (UID: \"8222de4a-dfe0-41eb-8857-142fa0f5919a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:31.114543 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:31.114470 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rp9dr\"" Jul 30 17:01:31.122462 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:31.122444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" Jul 30 17:01:31.245193 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:31.245159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m"] Jul 30 17:01:31.650278 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:31.650246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:32.271064 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:32.271019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" event={"ID":"8222de4a-dfe0-41eb-8857-142fa0f5919a","Type":"ContainerStarted","Data":"41315c27782292140164d5762e100fd4a3fec1e50087fdba28e330f6810df385"} Jul 30 17:01:32.276270 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:32.276237 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:32.375705 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:32.375658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:32.477207 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:32.477165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:33.022852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:33.022823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:33.275852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:33.275762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" event={"ID":"8222de4a-dfe0-41eb-8857-142fa0f5919a","Type":"ContainerStarted","Data":"c63bfc30e7df036881e62b2b6458ef9d218b3d49ce03852bdd7233e697fc9cec"} Jul 30 17:01:33.275852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:33.275830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" event={"ID":"8222de4a-dfe0-41eb-8857-142fa0f5919a","Type":"ContainerStarted","Data":"7be3e24402502c64ba8147ff0df59dc90cc23e0edc9a1421220dcaf7d61cdfcb"} Jul 30 17:01:33.295286 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:33.295229 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p472m" podStartSLOduration=16.566811429 podStartE2EDuration="18.295207351s" totalImagesPullingTime="1.728395922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:33.294560449 +0000 UTC m=+62.967757927" watchObservedRunningTime="2026-07-30 17:01:33.295207351 +0000 UTC m=+62.968404829" Jul 30 17:01:33.296747 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:33.296720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:33.838958 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:33.838926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:34.019807 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:34.019770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:34.420636 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:34.420579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:35.016132 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:35.016100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:35.337264 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:35.337183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:35.437922 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:35.437885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:35.629064 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:35.628978 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:36.367553 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.367514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:36.467614 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.467583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:36.556656 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.556632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:36.556757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.556684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:36.559584 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.559567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 17:01:36.560584 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.560570 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 17:01:36.569733 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.569707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd1a463-eb1a-4afb-aba9-ea8294291672-metrics-certs\") pod \"network-metrics-daemon-jt82f\" (UID: \"6bd1a463-eb1a-4afb-aba9-ea8294291672\") " pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:36.569984 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.569969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/479c1d63-717e-42f7-b325-e3585799cd58-original-pull-secret\") pod \"global-pull-secret-syncer-9wmwb\" (UID: \"479c1d63-717e-42f7-b325-e3585799cd58\") " pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:36.623383 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.623320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:36.655332 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.655316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9wmwb" Jul 30 17:01:36.664006 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.663990 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dnsfv\"" Jul 30 17:01:36.672076 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.672048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jt82f" Jul 30 17:01:36.758986 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.758878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:36.762160 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.762124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2b8b\" (UniqueName: \"kubernetes.io/projected/47edcc66-8e41-41b6-9bb3-c457deebbda9-kube-api-access-x2b8b\") pod \"network-check-target-rsvsz\" (UID: \"47edcc66-8e41-41b6-9bb3-c457deebbda9\") " pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:36.809302 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.809256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9wmwb"] Jul 30 17:01:36.811876 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:36.811849 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479c1d63_717e_42f7_b325_e3585799cd58.slice/crio-812cf73aeb559a140ffba1d9776ba5989f1e1df93c2191e296c533456b0e9b1a WatchSource:0}: Error finding container 812cf73aeb559a140ffba1d9776ba5989f1e1df93c2191e296c533456b0e9b1a: Status 404 returned error can't find the container with id 812cf73aeb559a140ffba1d9776ba5989f1e1df93c2191e296c533456b0e9b1a Jul 30 17:01:36.838274 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.838253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jt82f"] Jul 30 17:01:36.841135 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:36.841115 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd1a463_eb1a_4afb_aba9_ea8294291672.slice/crio-bf2241362b4c6ef4d1fb0750283974ea261be7e0e47737ec696bbca9581869e4 WatchSource:0}: Error finding container bf2241362b4c6ef4d1fb0750283974ea261be7e0e47737ec696bbca9581869e4: Status 404 returned error can't find the container with id bf2241362b4c6ef4d1fb0750283974ea261be7e0e47737ec696bbca9581869e4 Jul 30 17:01:36.951578 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.951558 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcvkj\"" Jul 30 17:01:36.959318 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:36.959300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:37.098905 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.098876 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rsvsz"] Jul 30 17:01:37.101548 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:37.101525 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47edcc66_8e41_41b6_9bb3_c457deebbda9.slice/crio-16b2d65080ed1945758251555c63336f39fbfb71414f7518480aff2470117ff0 WatchSource:0}: Error finding container 16b2d65080ed1945758251555c63336f39fbfb71414f7518480aff2470117ff0: Status 404 returned error can't find the container with id 16b2d65080ed1945758251555c63336f39fbfb71414f7518480aff2470117ff0 Jul 30 17:01:37.289274 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.289244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jt82f" event={"ID":"6bd1a463-eb1a-4afb-aba9-ea8294291672","Type":"ContainerStarted","Data":"bf2241362b4c6ef4d1fb0750283974ea261be7e0e47737ec696bbca9581869e4"} Jul 30 17:01:37.290426 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.290404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9wmwb" event={"ID":"479c1d63-717e-42f7-b325-e3585799cd58","Type":"ContainerStarted","Data":"812cf73aeb559a140ffba1d9776ba5989f1e1df93c2191e296c533456b0e9b1a"} Jul 30 17:01:37.291753 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.291732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rsvsz" event={"ID":"47edcc66-8e41-41b6-9bb3-c457deebbda9","Type":"ContainerStarted","Data":"79d86096ae0f72e916c8b42754ac07e148b865171579e73c46fd6768cd94160e"} Jul 30 17:01:37.291823 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.291768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rsvsz" event={"ID":"47edcc66-8e41-41b6-9bb3-c457deebbda9","Type":"ContainerStarted","Data":"16b2d65080ed1945758251555c63336f39fbfb71414f7518480aff2470117ff0"} Jul 30 17:01:37.291864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.291849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:01:37.309903 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.309859 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rsvsz" podStartSLOduration=66.309842323 podStartE2EDuration="1m6.309842323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:37.309240364 +0000 UTC m=+66.982437840" watchObservedRunningTime="2026-07-30 17:01:37.309842323 +0000 UTC m=+66.983039801" Jul 30 17:01:37.567701 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.567550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:37.570261 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.570236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/643e3b9d-d9a0-401d-bc08-0ba5554bbba8-metrics-tls\") pod \"dns-default-2twfz\" (UID: \"643e3b9d-d9a0-401d-bc08-0ba5554bbba8\") " pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:37.796756 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.796724 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bcbk6\"" Jul 30 17:01:37.804456 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.804217 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:37.980353 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:37.980320 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2twfz"] Jul 30 17:01:37.983888 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:37.983865 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643e3b9d_d9a0_401d_bc08_0ba5554bbba8.slice/crio-fa1935bf2c81cfa5728d3c6b99a381723d01647918f2430c9e7cef14beb7f532 WatchSource:0}: Error finding container fa1935bf2c81cfa5728d3c6b99a381723d01647918f2430c9e7cef14beb7f532: Status 404 returned error can't find the container with id fa1935bf2c81cfa5728d3c6b99a381723d01647918f2430c9e7cef14beb7f532 Jul 30 17:01:38.296981 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:38.296935 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2twfz" event={"ID":"643e3b9d-d9a0-401d-bc08-0ba5554bbba8","Type":"ContainerStarted","Data":"fa1935bf2c81cfa5728d3c6b99a381723d01647918f2430c9e7cef14beb7f532"} Jul 30 17:01:38.564199 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:38.564107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:38.630450 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:38.630403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:38.981588 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:38.981555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:38.981758 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:38.981598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:38.982138 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:38.982091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:38.985104 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:38.985085 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eaca9fc-0810-4b49-9e7b-b887e6407fc1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ppfb\" (UID: \"6eaca9fc-0810-4b49-9e7b-b887e6407fc1\") " pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:39.065079 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:39.065049 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brp5b\"" Jul 30 17:01:39.072506 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:39.072437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ppfb" Jul 30 17:01:39.239581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:39.239535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ppfb"] Jul 30 17:01:39.243795 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:39.243770 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaca9fc_0810_4b49_9e7b_b887e6407fc1.slice/crio-3817a07419003abe45026dab3d3b452da6978ee5527638d96d8d9595cbb8b619 WatchSource:0}: Error finding container 3817a07419003abe45026dab3d3b452da6978ee5527638d96d8d9595cbb8b619: Status 404 returned error can't find the container with id 3817a07419003abe45026dab3d3b452da6978ee5527638d96d8d9595cbb8b619 Jul 30 17:01:39.302948 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:39.302910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ppfb" event={"ID":"6eaca9fc-0810-4b49-9e7b-b887e6407fc1","Type":"ContainerStarted","Data":"3817a07419003abe45026dab3d3b452da6978ee5527638d96d8d9595cbb8b619"} Jul 30 17:01:39.304812 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:39.304785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jt82f" event={"ID":"6bd1a463-eb1a-4afb-aba9-ea8294291672","Type":"ContainerStarted","Data":"59b325e10b69579188a97a0e26c9b57ff62c164bca34d76a90af96b79f17f26a"} Jul 30 17:01:39.304956 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:39.304831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jt82f" event={"ID":"6bd1a463-eb1a-4afb-aba9-ea8294291672","Type":"ContainerStarted","Data":"a0f94ae423aa8880cb8fcb34112c6b11192c7493009dcf4ee46d0d05dcf7c936"} Jul 30 17:01:39.323488 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:39.323389 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jt82f" podStartSLOduration=66.498948517 podStartE2EDuration="1m8.323357026s" totalImagesPullingTime="1.824408509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:39.321911564 +0000 UTC m=+68.995109041" watchObservedRunningTime="2026-07-30 17:01:39.323357026 +0000 UTC m=+68.996554508" Jul 30 17:01:40.335458 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:40.335419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:40.436586 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:40.436550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:40.538141 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:40.538109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:40.638273 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:40.638182 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:41.245910 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:41.245782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:41.324922 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:41.323949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9wmwb" event={"ID":"479c1d63-717e-42f7-b325-e3585799cd58","Type":"ContainerStarted","Data":"491c0fb7e75e15116fc79ce6f75c4b1fe48be9c3a368a4acda7e087ba17bd248"} Jul 30 17:01:41.332217 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:41.332184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2twfz" event={"ID":"643e3b9d-d9a0-401d-bc08-0ba5554bbba8","Type":"ContainerStarted","Data":"5c21e7561defdfbdfc1370a4a9871254a20905919f5a483e73bb606f03344eb0"} Jul 30 17:01:41.344618 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:41.344284 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9wmwb" podStartSLOduration=65.997382062 podStartE2EDuration="1m10.344265063s" totalImagesPullingTime="4.346883001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:41.342944139 +0000 UTC m=+71.016141617" watchObservedRunningTime="2026-07-30 17:01:41.344265063 +0000 UTC m=+71.017462541" Jul 30 17:01:41.346552 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:41.345721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:41.446854 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:41.446773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:41.513039 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:41.512996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:42.030953 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:42.030903 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:42.225293 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:42.225245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:42.338608 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:42.338521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2twfz" event={"ID":"643e3b9d-d9a0-401d-bc08-0ba5554bbba8","Type":"ContainerStarted","Data":"2756c040c29b784a4476b376b0fbf3eff735d67308ff6fb4b961653049eb1a5f"} Jul 30 17:01:42.358479 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:42.358428 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2twfz" podStartSLOduration=34.190002145 podStartE2EDuration="37.35841054s" totalImagesPullingTime="3.168408395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:42.357147567 +0000 UTC m=+72.030345044" watchObservedRunningTime="2026-07-30 17:01:42.35841054 +0000 UTC m=+72.031608017" Jul 30 17:01:42.696119 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:42.696042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:43.342951 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:43.342915 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:43.450301 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:43.450270 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:43.514352 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:43.514319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:45.350477 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.350436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:45.421083 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.421044 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6875b55898-drb6h"] Jul 30 17:01:45.423101 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.423074 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-dw4st"] Jul 30 17:01:45.423238 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.423190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.425160 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.425141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dw4st" Jul 30 17:01:45.426472 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.426450 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 17:01:45.426729 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.426706 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 17:01:45.427053 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.427031 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pn7tb\"" Jul 30 17:01:45.429854 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.429794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 17:01:45.431413 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.431197 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7wl8p\"" Jul 30 17:01:45.431413 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.431266 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 30 17:01:45.431919 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.431652 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 30 17:01:45.435920 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.435901 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 17:01:45.441227 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.441150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dw4st"] Jul 30 17:01:45.442305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.442285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6875b55898-drb6h"] Jul 30 17:01:45.463021 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.462994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:45.532107 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e0e83a3-3106-4899-9221-b31add3218ce-installation-pull-secrets\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532303 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e0e83a3-3106-4899-9221-b31add3218ce-registry-certificates\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532303 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e83a3-3106-4899-9221-b31add3218ce-trusted-ca\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532413 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6zd\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-kube-api-access-hp6zd\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532413 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e0e83a3-3106-4899-9221-b31add3218ce-image-registry-private-configuration\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532516 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-registry-tls\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532516 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e0e83a3-3106-4899-9221-b31add3218ce-ca-trust-extracted\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532516 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-bound-sa-token\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.532658 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.532544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqvw\" (UniqueName: \"kubernetes.io/projected/0d6a813e-46a1-4727-8eeb-362e745cd45d-kube-api-access-bnqvw\") pod \"downloads-7c97dc4555-dw4st\" (UID: \"0d6a813e-46a1-4727-8eeb-362e745cd45d\") " pod="openshift-console/downloads-7c97dc4555-dw4st" Jul 30 17:01:45.559465 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.559423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:45.632995 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.632915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqvw\" (UniqueName: \"kubernetes.io/projected/0d6a813e-46a1-4727-8eeb-362e745cd45d-kube-api-access-bnqvw\") pod \"downloads-7c97dc4555-dw4st\" (UID: \"0d6a813e-46a1-4727-8eeb-362e745cd45d\") " pod="openshift-console/downloads-7c97dc4555-dw4st" Jul 30 17:01:45.632995 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.632957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e0e83a3-3106-4899-9221-b31add3218ce-installation-pull-secrets\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e0e83a3-3106-4899-9221-b31add3218ce-registry-certificates\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e83a3-3106-4899-9221-b31add3218ce-trusted-ca\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6zd\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-kube-api-access-hp6zd\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633330 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e0e83a3-3106-4899-9221-b31add3218ce-image-registry-private-configuration\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633330 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-registry-tls\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633330 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e0e83a3-3106-4899-9221-b31add3218ce-ca-trust-extracted\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633330 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-bound-sa-token\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.633809 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e0e83a3-3106-4899-9221-b31add3218ce-ca-trust-extracted\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.634029 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.633998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e0e83a3-3106-4899-9221-b31add3218ce-registry-certificates\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.634148 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.634051 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e83a3-3106-4899-9221-b31add3218ce-trusted-ca\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.635944 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.635916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e0e83a3-3106-4899-9221-b31add3218ce-installation-pull-secrets\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.636064 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.636046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e0e83a3-3106-4899-9221-b31add3218ce-image-registry-private-configuration\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.636654 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.636635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-registry-tls\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.645151 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.645113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6zd\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-kube-api-access-hp6zd\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.645539 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.645519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqvw\" (UniqueName: \"kubernetes.io/projected/0d6a813e-46a1-4727-8eeb-362e745cd45d-kube-api-access-bnqvw\") pod \"downloads-7c97dc4555-dw4st\" (UID: \"0d6a813e-46a1-4727-8eeb-362e745cd45d\") " pod="openshift-console/downloads-7c97dc4555-dw4st" Jul 30 17:01:45.645897 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.645878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e0e83a3-3106-4899-9221-b31add3218ce-bound-sa-token\") pod \"image-registry-6875b55898-drb6h\" (UID: \"8e0e83a3-3106-4899-9221-b31add3218ce\") " pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.659788 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.659759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:45.734719 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.734505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:45.741793 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.741777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dw4st" Jul 30 17:01:45.765871 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.765819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:45.906806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.906778 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6875b55898-drb6h"] Jul 30 17:01:45.909044 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:45.909021 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0e83a3_3106_4899_9221_b31add3218ce.slice/crio-4f309ffe95c2eaeb89c5c493c2011e8136583cf4aeb61f6263b46db2841d559f WatchSource:0}: Error finding container 4f309ffe95c2eaeb89c5c493c2011e8136583cf4aeb61f6263b46db2841d559f: Status 404 returned error can't find the container with id 4f309ffe95c2eaeb89c5c493c2011e8136583cf4aeb61f6263b46db2841d559f Jul 30 17:01:45.921431 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:45.921406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dw4st"] Jul 30 17:01:45.923938 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:45.923914 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6a813e_46a1_4727_8eeb_362e745cd45d.slice/crio-c4e6d6a577a9a0e3a0ec67f6cc95aa5d2ecca3327dbb8733c66e9152fe6e9edc WatchSource:0}: Error finding container c4e6d6a577a9a0e3a0ec67f6cc95aa5d2ecca3327dbb8733c66e9152fe6e9edc: Status 404 returned error can't find the container with id c4e6d6a577a9a0e3a0ec67f6cc95aa5d2ecca3327dbb8733c66e9152fe6e9edc Jul 30 17:01:46.356280 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:46.356233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ppfb" event={"ID":"6eaca9fc-0810-4b49-9e7b-b887e6407fc1","Type":"ContainerStarted","Data":"edb8d34c116dcbc75c6bb53c2d5c6a0a1e52c2b871dd6ff98b224f7af2d22a90"} Jul 30 17:01:46.357655 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:46.357623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dw4st" event={"ID":"0d6a813e-46a1-4727-8eeb-362e745cd45d","Type":"ContainerStarted","Data":"c4e6d6a577a9a0e3a0ec67f6cc95aa5d2ecca3327dbb8733c66e9152fe6e9edc"} Jul 30 17:01:46.359786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:46.359714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6875b55898-drb6h" event={"ID":"8e0e83a3-3106-4899-9221-b31add3218ce","Type":"ContainerStarted","Data":"6674145ec2cc1fedfc69b766b1e5735b4fe530ad3551c0fce7a35c10387c9d22"} Jul 30 17:01:46.359786 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:46.359757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6875b55898-drb6h" event={"ID":"8e0e83a3-3106-4899-9221-b31add3218ce","Type":"ContainerStarted","Data":"4f309ffe95c2eaeb89c5c493c2011e8136583cf4aeb61f6263b46db2841d559f"} Jul 30 17:01:46.359956 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:46.359920 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:01:46.384061 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:46.383871 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6875b55898-drb6h" podStartSLOduration=1.383856974 podStartE2EDuration="1.383856974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:46.382817243 +0000 UTC m=+76.056014716" watchObservedRunningTime="2026-07-30 17:01:46.383856974 +0000 UTC m=+76.057054451" Jul 30 17:01:46.923910 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:46.923872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:47.072526 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:47.072306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:47.150107 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:47.150069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:48.853422 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:48.853361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:49.880887 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:49.880852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:49.938645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:49.938586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:50.379006 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:50.378967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ppfb" event={"ID":"6eaca9fc-0810-4b49-9e7b-b887e6407fc1","Type":"ContainerStarted","Data":"30af921146045516728cfa9178d2d6926a84a0e63f99635e44da1d2256254847"} Jul 30 17:01:50.458549 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:50.458505 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:50.524610 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:50.524575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:51.864527 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.864490 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7696fdb7bb-nftqx"] Jul 30 17:01:51.887020 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.886988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7696fdb7bb-nftqx"] Jul 30 17:01:51.887197 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.887128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:51.889944 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.889892 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 30 17:01:51.891256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.890998 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wrh2v\"" Jul 30 17:01:51.891256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.891009 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 30 17:01:51.891256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.891071 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 30 17:01:51.891256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.891134 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 30 17:01:51.891256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.891203 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 30 17:01:51.991138 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.991100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74p8r\" (UniqueName: \"kubernetes.io/projected/6f1598f9-81bd-4742-9520-6c51417cc888-kube-api-access-74p8r\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:51.991138 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.991146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-oauth-serving-cert\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:51.991429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.991247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-console-config\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:51.991429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.991303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-serving-cert\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:51.991429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.991335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-service-ca\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:51.991429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:51.991395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-oauth-config\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.093202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.093151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-serving-cert\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.093202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.093201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-service-ca\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.093480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.093248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-oauth-config\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.093480 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.093276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p8r\" (UniqueName: \"kubernetes.io/projected/6f1598f9-81bd-4742-9520-6c51417cc888-kube-api-access-74p8r\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.094479 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.093705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-oauth-serving-cert\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.094479 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.093794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-console-config\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.094479 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.094081 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-service-ca\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.094479 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.094419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-console-config\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.094860 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.094652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-oauth-serving-cert\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.096961 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.096934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-oauth-config\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.097092 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.097077 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-serving-cert\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.102481 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.102436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p8r\" (UniqueName: \"kubernetes.io/projected/6f1598f9-81bd-4742-9520-6c51417cc888-kube-api-access-74p8r\") pod \"console-7696fdb7bb-nftqx\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.200054 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.199825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:01:52.253242 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.253206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:52.382191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:52.382155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7696fdb7bb-nftqx"] Jul 30 17:01:53.348690 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:53.348144 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2twfz" Jul 30 17:01:53.462313 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:53.462253 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:53.519085 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:53.518999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:54.221562 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:01:54.221520 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1598f9_81bd_4742_9520_6c51417cc888.slice/crio-d4a8613d47ce521c9c715ed95517121d05b73dd17e74b812c6bcdc4b0721d2d5 WatchSource:0}: Error finding container d4a8613d47ce521c9c715ed95517121d05b73dd17e74b812c6bcdc4b0721d2d5: Status 404 returned error can't find the container with id d4a8613d47ce521c9c715ed95517121d05b73dd17e74b812c6bcdc4b0721d2d5 Jul 30 17:01:54.399126 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:54.399083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ppfb" event={"ID":"6eaca9fc-0810-4b49-9e7b-b887e6407fc1","Type":"ContainerStarted","Data":"6ef0474717231fab7a831050bc12ad3f4ba92ec3645297dd82a1f3d83d9181b5"} Jul 30 17:01:54.400450 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:54.400424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7696fdb7bb-nftqx" event={"ID":"6f1598f9-81bd-4742-9520-6c51417cc888","Type":"ContainerStarted","Data":"d4a8613d47ce521c9c715ed95517121d05b73dd17e74b812c6bcdc4b0721d2d5"} Jul 30 17:01:54.419971 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:54.419914 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2ppfb" podStartSLOduration=16.560637321 podStartE2EDuration="31.419894042s" totalImagesPullingTime="14.859256721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:01:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:54.417745335 +0000 UTC m=+84.090942809" watchObservedRunningTime="2026-07-30 17:01:54.419894042 +0000 UTC m=+84.093091522" Jul 30 17:01:54.845662 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:54.845584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:54.899630 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:54.899583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:56.476159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:56.476118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:56.554186 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:56.554151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:57.044235 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.044197 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cfc4576cf-kcfv8"] Jul 30 17:01:57.046730 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.046709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.055251 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.055231 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 30 17:01:57.058062 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.058038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cfc4576cf-kcfv8"] Jul 30 17:01:57.139181 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.139144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-config\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.139365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.139187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czckg\" (UniqueName: \"kubernetes.io/projected/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-kube-api-access-czckg\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.139365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.139231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-trusted-ca-bundle\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.139365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.139272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-oauth-serving-cert\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.139592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.139389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-serving-cert\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.139592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.139432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-service-ca\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.139592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.139466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-oauth-config\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.240146 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.240103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czckg\" (UniqueName: \"kubernetes.io/projected/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-kube-api-access-czckg\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.240346 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.240160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-trusted-ca-bundle\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.240346 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.240201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-oauth-serving-cert\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.240346 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.240230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-serving-cert\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.240346 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.240263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-service-ca\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.240346 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.240300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-oauth-config\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.240346 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.240340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-config\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.241147 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.241094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-oauth-serving-cert\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.241257 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.241206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-config\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.241319 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.241251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-trusted-ca-bundle\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.241602 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.241582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-service-ca\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.243118 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.243093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-serving-cert\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.243222 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.243191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-oauth-config\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.251359 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.251335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czckg\" (UniqueName: \"kubernetes.io/projected/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-kube-api-access-czckg\") pod \"console-6cfc4576cf-kcfv8\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.268834 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.268797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:57.332839 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.332730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:57.358761 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.358725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:01:57.728235 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.728187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:57.827512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.827482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:57.926294 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:57.926215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:58.262322 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:58.262286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:58.464841 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:58.464805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:01:58.537562 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:01:58.537469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:01.261060 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:01.261021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:01.364996 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:01.364907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:02.916113 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:02.916079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:02.996821 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:02.996783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:03.218396 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:03.218344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:03.327029 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:03.326998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:03.426796 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:03.426749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:04.685807 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:04.685776 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cfc4576cf-kcfv8"] Jul 30 17:02:04.688467 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:02:04.688441 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d414bf_7d12_4886_ab57_d96d92b4d9e6.slice/crio-6477ebc255259055817daa079c8958f49b5daee1ab1c9049a487cb0d90b2014c WatchSource:0}: Error finding container 6477ebc255259055817daa079c8958f49b5daee1ab1c9049a487cb0d90b2014c: Status 404 returned error can't find the container with id 6477ebc255259055817daa079c8958f49b5daee1ab1c9049a487cb0d90b2014c Jul 30 17:02:05.447391 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.447331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dw4st" event={"ID":"0d6a813e-46a1-4727-8eeb-362e745cd45d","Type":"ContainerStarted","Data":"1e4de068155757b9cbb1fdb190a85d39d9d197ed2274acca15de1f83bfd5d450"} Jul 30 17:02:05.447593 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.447537 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-dw4st" Jul 30 17:02:05.449523 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.449488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7696fdb7bb-nftqx" event={"ID":"6f1598f9-81bd-4742-9520-6c51417cc888","Type":"ContainerStarted","Data":"49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3"} Jul 30 17:02:05.451368 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.451343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfc4576cf-kcfv8" event={"ID":"a9d414bf-7d12-4886-ab57-d96d92b4d9e6","Type":"ContainerStarted","Data":"3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8"} Jul 30 17:02:05.451481 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.451407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfc4576cf-kcfv8" event={"ID":"a9d414bf-7d12-4886-ab57-d96d92b4d9e6","Type":"ContainerStarted","Data":"6477ebc255259055817daa079c8958f49b5daee1ab1c9049a487cb0d90b2014c"} Jul 30 17:02:05.454023 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.453996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:05.461697 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.461678 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-dw4st" Jul 30 17:02:05.486039 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.485972 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-dw4st" podStartSLOduration=1.81287917 podStartE2EDuration="20.485952581s" totalImagesPullingTime="18.673073411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:05.467324565 +0000 UTC m=+95.140522043" watchObservedRunningTime="2026-07-30 17:02:05.485952581 +0000 UTC m=+95.159150058" Jul 30 17:02:05.503951 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.503884 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cfc4576cf-kcfv8" podStartSLOduration=8.503861597 podStartE2EDuration="8.503861597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:05.503262652 +0000 UTC m=+95.176460130" watchObservedRunningTime="2026-07-30 17:02:05.503861597 +0000 UTC m=+95.177059074" Jul 30 17:02:05.524041 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.523979 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7696fdb7bb-nftqx" podStartSLOduration=4.205741108 podStartE2EDuration="14.523957742s" totalImagesPullingTime="10.318216634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:05.522534622 +0000 UTC m=+95.195732099" watchObservedRunningTime="2026-07-30 17:02:05.523957742 +0000 UTC m=+95.197155222" Jul 30 17:02:05.563813 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.563773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:05.665637 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:05.665601 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:07.358993 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.358946 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:02:07.359496 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.359172 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:02:07.364845 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.364818 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:02:07.371043 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.371015 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6875b55898-drb6h" Jul 30 17:02:07.465302 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.465270 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:02:07.521729 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.521688 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7696fdb7bb-nftqx"] Jul 30 17:02:07.534436 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.534394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:07.631055 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.630959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:07.730162 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.730129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:07.828237 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:07.828201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.061983 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.061931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.166030 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.165991 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.265929 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.265891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.300189 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.300155 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rsvsz" Jul 30 17:02:08.317135 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.316887 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c"] Jul 30 17:02:08.337023 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.336983 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c"] Jul 30 17:02:08.337445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.337430 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.341008 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.340989 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xq4vd\"" Jul 30 17:02:08.342306 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.342233 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 30 17:02:08.343465 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.343441 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 17:02:08.343781 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.343762 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 30 17:02:08.344161 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.344022 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 17:02:08.344434 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.343605 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 17:02:08.360329 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.360205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.366907 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.366881 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jsl2b"] Jul 30 17:02:08.381513 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.381488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.396449 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.396430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 17:02:08.397732 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.396645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 17:02:08.398115 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.398101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kdk5k\"" Jul 30 17:02:08.398502 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.396745 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/620e2f01-559e-4eb2-9734-fc4721e2c53e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8k6n\" (UniqueName: \"kubernetes.io/projected/620e2f01-559e-4eb2-9734-fc4721e2c53e-kube-api-access-w8k6n\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-sys\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-root\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgj6b\" (UniqueName: \"kubernetes.io/projected/02922d64-a0c1-4fc4-91a8-b65655d2e369-kube-api-access-jgj6b\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-wtmp\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02922d64-a0c1-4fc4-91a8-b65655d2e369-metrics-client-ca\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.439393 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.431467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-textfile\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.447387 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.445249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.532347 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.532306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-wtmp\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.532581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.532512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-wtmp\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.532657 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.532636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.532844 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.532829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02922d64-a0c1-4fc4-91a8-b65655d2e369-metrics-client-ca\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.532955 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.532942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-textfile\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.533110 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/620e2f01-559e-4eb2-9734-fc4721e2c53e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.533208 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8k6n\" (UniqueName: \"kubernetes.io/projected/620e2f01-559e-4eb2-9734-fc4721e2c53e-kube-api-access-w8k6n\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.533297 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-sys\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.533429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.533556 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02922d64-a0c1-4fc4-91a8-b65655d2e369-metrics-client-ca\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.533698 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-root\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.533799 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.533933 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.533917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.534068 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.534037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.534169 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.534156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgj6b\" (UniqueName: \"kubernetes.io/projected/02922d64-a0c1-4fc4-91a8-b65655d2e369-kube-api-access-jgj6b\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.534316 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.534294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/620e2f01-559e-4eb2-9734-fc4721e2c53e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.534615 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.534594 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-textfile\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.535086 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.535064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-sys\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.535645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.535213 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02922d64-a0c1-4fc4-91a8-b65655d2e369-root\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.535778 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:08.535247 2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 30 17:02:08.535879 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.535857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-accelerators-collector-config\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.536021 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:08.536011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-tls podName:620e2f01-559e-4eb2-9734-fc4721e2c53e nodeName:}" failed. No retries permitted until 2026-07-30 17:02:09.035978212 +0000 UTC m=+98.709175675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-zc28c" (UID: "620e2f01-559e-4eb2-9734-fc4721e2c53e") : secret "openshift-state-metrics-tls" not found Jul 30 17:02:08.536297 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:08.535283 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 30 17:02:08.536451 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:08.536442 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls podName:02922d64-a0c1-4fc4-91a8-b65655d2e369 nodeName:}" failed. No retries permitted until 2026-07-30 17:02:09.036426186 +0000 UTC m=+98.709623645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls") pod "node-exporter-jsl2b" (UID: "02922d64-a0c1-4fc4-91a8-b65655d2e369") : secret "node-exporter-tls" not found Jul 30 17:02:08.536542 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.536008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.546591 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.546546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.550804 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.550740 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8k6n\" (UniqueName: \"kubernetes.io/projected/620e2f01-559e-4eb2-9734-fc4721e2c53e-kube-api-access-w8k6n\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:08.552316 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.552294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgj6b\" (UniqueName: \"kubernetes.io/projected/02922d64-a0c1-4fc4-91a8-b65655d2e369-kube-api-access-jgj6b\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:08.597415 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.597301 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.695149 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.695115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:08.797363 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:08.797328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:09.001358 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.001324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:09.039792 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.039753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:09.040107 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.040081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:09.040231 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:09.040208 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 30 17:02:09.040294 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:09.040284 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls podName:02922d64-a0c1-4fc4-91a8-b65655d2e369 nodeName:}" failed. No retries permitted until 2026-07-30 17:02:10.04026489 +0000 UTC m=+99.713462349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls") pod "node-exporter-jsl2b" (UID: "02922d64-a0c1-4fc4-91a8-b65655d2e369") : secret "node-exporter-tls" not found Jul 30 17:02:09.042819 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.042776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/620e2f01-559e-4eb2-9734-fc4721e2c53e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-zc28c\" (UID: \"620e2f01-559e-4eb2-9734-fc4721e2c53e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:09.116575 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.116499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:09.215204 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.215161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:09.257047 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.256961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" Jul 30 17:02:09.317680 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.317623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:09.411065 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.411031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:09.417256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.417117 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 17:02:09.447321 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.447287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 17:02:09.447528 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.447463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.450516 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.450489 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.450732 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.450912 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.451267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.451528 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.451733 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.451942 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-27nc2\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.452119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.452151 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 17:02:09.452305 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.452311 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 17:02:09.457065 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.456321 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c"] Jul 30 17:02:09.459655 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:02:09.459603 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620e2f01_559e_4eb2_9734_fc4721e2c53e.slice/crio-a201cee5e13fd3d82d005587828b51555e17c5ae19c0418d5ba0230aea89cde1 WatchSource:0}: Error finding container a201cee5e13fd3d82d005587828b51555e17c5ae19c0418d5ba0230aea89cde1: Status 404 returned error can't find the container with id a201cee5e13fd3d82d005587828b51555e17c5ae19c0418d5ba0230aea89cde1 Jul 30 17:02:09.470715 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.470690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" event={"ID":"620e2f01-559e-4eb2-9734-fc4721e2c53e","Type":"ContainerStarted","Data":"a201cee5e13fd3d82d005587828b51555e17c5ae19c0418d5ba0230aea89cde1"} Jul 30 17:02:09.495281 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.495250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:09.546308 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.546246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.546693 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.546369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fc407a1-e51b-4652-a677-935c67c6a364-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.546806 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.546753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc407a1-e51b-4652-a677-935c67c6a364-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.546954 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.546924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547120 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fc407a1-e51b-4652-a677-935c67c6a364-config-out\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547237 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85wb\" (UniqueName: \"kubernetes.io/projected/9fc407a1-e51b-4652-a677-935c67c6a364-kube-api-access-j85wb\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547413 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547513 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547574 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547574 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fc407a1-e51b-4652-a677-935c67c6a364-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547733 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547733 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc407a1-e51b-4652-a677-935c67c6a364-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.547733 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.547719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-web-config\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.648791 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.648751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-web-config\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.648937 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.648816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.648937 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.648851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fc407a1-e51b-4652-a677-935c67c6a364-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.648937 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.648880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc407a1-e51b-4652-a677-935c67c6a364-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.648941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.648966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fc407a1-e51b-4652-a677-935c67c6a364-config-out\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.648990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85wb\" (UniqueName: \"kubernetes.io/projected/9fc407a1-e51b-4652-a677-935c67c6a364-kube-api-access-j85wb\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.649040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.649080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649338 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.649112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649338 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.649140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fc407a1-e51b-4652-a677-935c67c6a364-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649338 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.649178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649338 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.649228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc407a1-e51b-4652-a677-935c67c6a364-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.649547 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:09.649515 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 30 17:02:09.649618 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:09.649600 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-main-tls podName:9fc407a1-e51b-4652-a677-935c67c6a364 nodeName:}" failed. No retries permitted until 2026-07-30 17:02:10.149579944 +0000 UTC m=+99.822777421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9fc407a1-e51b-4652-a677-935c67c6a364") : secret "alertmanager-main-tls" not found Jul 30 17:02:09.652028 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.650529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc407a1-e51b-4652-a677-935c67c6a364-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.652028 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.650783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fc407a1-e51b-4652-a677-935c67c6a364-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.652028 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.651479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fc407a1-e51b-4652-a677-935c67c6a364-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.653084 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.652627 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.653084 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.653037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fc407a1-e51b-4652-a677-935c67c6a364-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.653738 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.653692 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.653970 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.653927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.654333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.654225 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.654504 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.654445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fc407a1-e51b-4652-a677-935c67c6a364-config-out\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.655384 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.655346 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-web-config\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.656120 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.656081 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.665896 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.665871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85wb\" (UniqueName: \"kubernetes.io/projected/9fc407a1-e51b-4652-a677-935c67c6a364-kube-api-access-j85wb\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:09.996860 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:09.996767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:10.053726 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.053680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:10.056548 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.056518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02922d64-a0c1-4fc4-91a8-b65655d2e369-node-exporter-tls\") pod \"node-exporter-jsl2b\" (UID: \"02922d64-a0c1-4fc4-91a8-b65655d2e369\") " pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:10.154796 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.154751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:10.157675 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.157644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fc407a1-e51b-4652-a677-935c67c6a364-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fc407a1-e51b-4652-a677-935c67c6a364\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:10.211819 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.211775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jsl2b" Jul 30 17:02:10.249244 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:02:10.249176 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02922d64_a0c1_4fc4_91a8_b65655d2e369.slice/crio-c32d95e716f808cf5df194fe29403e8f0134aae6db91ff83861f9182f9c62488 WatchSource:0}: Error finding container c32d95e716f808cf5df194fe29403e8f0134aae6db91ff83861f9182f9c62488: Status 404 returned error can't find the container with id c32d95e716f808cf5df194fe29403e8f0134aae6db91ff83861f9182f9c62488 Jul 30 17:02:10.364351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.364279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 17:02:10.478592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.478537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" event={"ID":"620e2f01-559e-4eb2-9734-fc4721e2c53e","Type":"ContainerStarted","Data":"2852ba3b3b8698a4f6f722c40f69ab836033aed358e544297e0b8edf6ee5ede7"} Jul 30 17:02:10.479179 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.478651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" event={"ID":"620e2f01-559e-4eb2-9734-fc4721e2c53e","Type":"ContainerStarted","Data":"900bff7a99f8e0392272514e4c905d6c8c27f9efe7efc236c9bf8ae62627dd98"} Jul 30 17:02:10.480795 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.480760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsl2b" event={"ID":"02922d64-a0c1-4fc4-91a8-b65655d2e369","Type":"ContainerStarted","Data":"c32d95e716f808cf5df194fe29403e8f0134aae6db91ff83861f9182f9c62488"} Jul 30 17:02:10.570712 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:10.570681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 17:02:10.572205 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:02:10.572174 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc407a1_e51b_4652_a677_935c67c6a364.slice/crio-79cc455ad1a9e76a71291753a949274716adb0a1e0121fa18c7917b37fb08e6a WatchSource:0}: Error finding container 79cc455ad1a9e76a71291753a949274716adb0a1e0121fa18c7917b37fb08e6a: Status 404 returned error can't find the container with id 79cc455ad1a9e76a71291753a949274716adb0a1e0121fa18c7917b37fb08e6a Jul 30 17:02:11.198474 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:11.198439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:11.307362 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:11.307328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:11.403983 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:11.403801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:11.490003 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:11.489893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerStarted","Data":"79cc455ad1a9e76a71291753a949274716adb0a1e0121fa18c7917b37fb08e6a"} Jul 30 17:02:11.760712 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:11.760618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:11.825972 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:11.825929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:12.200298 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:12.200266 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:02:12.496779 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:12.496694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" event={"ID":"620e2f01-559e-4eb2-9734-fc4721e2c53e","Type":"ContainerStarted","Data":"6f918d7ed4041b7e491aad0582420d4b5248d6862868c2def43e650bf7f3bdb4"} Jul 30 17:02:12.516521 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:12.516457 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-zc28c" podStartSLOduration=2.889298051 podStartE2EDuration="4.516439617s" totalImagesPullingTime="1.627141566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:12.51591039 +0000 UTC m=+102.189107867" watchObservedRunningTime="2026-07-30 17:02:12.516439617 +0000 UTC m=+102.189637093" Jul 30 17:02:13.101534 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.101085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:13.124003 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.123967 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd"] Jul 30 17:02:13.135451 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.135416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.138878 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.138813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd"] Jul 30 17:02:13.139173 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.139149 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c495s\"" Jul 30 17:02:13.139736 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.139359 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 30 17:02:13.170566 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.170470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:13.186832 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.186799 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c19a42a8-010d-4954-8b4e-af3e11414ce8-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jwfzd\" (UID: \"c19a42a8-010d-4954-8b4e-af3e11414ce8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.186995 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.186870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk87r\" (UniqueName: \"kubernetes.io/projected/c19a42a8-010d-4954-8b4e-af3e11414ce8-kube-api-access-lk87r\") pod \"monitoring-plugin-68b7fc9574-jwfzd\" (UID: \"c19a42a8-010d-4954-8b4e-af3e11414ce8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.288405 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.288345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c19a42a8-010d-4954-8b4e-af3e11414ce8-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jwfzd\" (UID: \"c19a42a8-010d-4954-8b4e-af3e11414ce8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.288606 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.288455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk87r\" (UniqueName: \"kubernetes.io/projected/c19a42a8-010d-4954-8b4e-af3e11414ce8-kube-api-access-lk87r\") pod \"monitoring-plugin-68b7fc9574-jwfzd\" (UID: \"c19a42a8-010d-4954-8b4e-af3e11414ce8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.290865 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.290843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c19a42a8-010d-4954-8b4e-af3e11414ce8-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jwfzd\" (UID: \"c19a42a8-010d-4954-8b4e-af3e11414ce8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.297273 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.297248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk87r\" (UniqueName: \"kubernetes.io/projected/c19a42a8-010d-4954-8b4e-af3e11414ce8-kube-api-access-lk87r\") pod \"monitoring-plugin-68b7fc9574-jwfzd\" (UID: \"c19a42a8-010d-4954-8b4e-af3e11414ce8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.314391 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.314333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:13.414900 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.414861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:13.455860 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.455829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:13.502841 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.502802 2576 generic.go:356] "Generic (PLEG): container finished" podID="02922d64-a0c1-4fc4-91a8-b65655d2e369" containerID="05c1347eab9add638d4020e470ede397740ce506651a2ef9ffcdeceb6202a51a" exitCode=0 Jul 30 17:02:13.503290 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.502896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsl2b" event={"ID":"02922d64-a0c1-4fc4-91a8-b65655d2e369","Type":"ContainerDied","Data":"05c1347eab9add638d4020e470ede397740ce506651a2ef9ffcdeceb6202a51a"} Jul 30 17:02:13.504787 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.504758 2576 generic.go:356] "Generic (PLEG): container finished" podID="9fc407a1-e51b-4652-a677-935c67c6a364" containerID="668b443579dcab8266c4e92487d76ad026ccdf4beb499e414c5acab727f796ce" exitCode=0 Jul 30 17:02:13.504951 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.504851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerDied","Data":"668b443579dcab8266c4e92487d76ad026ccdf4beb499e414c5acab727f796ce"} Jul 30 17:02:13.569592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.569565 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-555c98fb9b-6w9nj"] Jul 30 17:02:13.592234 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.592200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-555c98fb9b-6w9nj"] Jul 30 17:02:13.592436 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.592419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.595042 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.595019 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 30 17:02:13.596145 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.596124 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7md4r\"" Jul 30 17:02:13.596361 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.596331 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 30 17:02:13.596709 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.596690 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 30 17:02:13.597084 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.597065 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 30 17:02:13.597253 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.597229 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 30 17:02:13.601759 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.601737 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 30 17:02:13.614915 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.614891 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd"] Jul 30 17:02:13.617305 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:02:13.617276 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19a42a8_010d_4954_8b4e_af3e11414ce8.slice/crio-24a1fb986d25312098da030ce6d478232fa12bec056ec69f39661616a2eb89fa WatchSource:0}: Error finding container 24a1fb986d25312098da030ce6d478232fa12bec056ec69f39661616a2eb89fa: Status 404 returned error can't find the container with id 24a1fb986d25312098da030ce6d478232fa12bec056ec69f39661616a2eb89fa Jul 30 17:02:13.691590 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-telemeter-client-tls\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.691742 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-metrics-client-ca\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.691742 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-secret-telemeter-client\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.691838 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqmn\" (UniqueName: \"kubernetes.io/projected/ad4fd300-c9e2-4760-902c-40af264af1f3-kube-api-access-8lqmn\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.691838 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-serving-certs-ca-bundle\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.691838 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.691838 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-federate-client-tls\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.691987 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.691848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793130 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-telemeter-client-tls\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793130 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-metrics-client-ca\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-secret-telemeter-client\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqmn\" (UniqueName: \"kubernetes.io/projected/ad4fd300-c9e2-4760-902c-40af264af1f3-kube-api-access-8lqmn\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-serving-certs-ca-bundle\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-federate-client-tls\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793649 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.793953 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.793927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-metrics-client-ca\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.794108 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.794070 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-serving-certs-ca-bundle\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.794287 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.794258 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad4fd300-c9e2-4760-902c-40af264af1f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.795929 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.795911 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-federate-client-tls\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.796180 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.795944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.796180 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.796088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-telemeter-client-tls\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.796180 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.796139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad4fd300-c9e2-4760-902c-40af264af1f3-secret-telemeter-client\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.801880 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.801859 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqmn\" (UniqueName: \"kubernetes.io/projected/ad4fd300-c9e2-4760-902c-40af264af1f3-kube-api-access-8lqmn\") pod \"telemeter-client-555c98fb9b-6w9nj\" (UID: \"ad4fd300-c9e2-4760-902c-40af264af1f3\") " pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:13.904311 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:13.904270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" Jul 30 17:02:14.056351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.056314 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-555c98fb9b-6w9nj"] Jul 30 17:02:14.059243 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:02:14.059212 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4fd300_c9e2_4760_902c_40af264af1f3.slice/crio-7b1c8bbc83a9160ee3addae328073266e4dbfbff43d9ba35f633cabc35385410 WatchSource:0}: Error finding container 7b1c8bbc83a9160ee3addae328073266e4dbfbff43d9ba35f633cabc35385410: Status 404 returned error can't find the container with id 7b1c8bbc83a9160ee3addae328073266e4dbfbff43d9ba35f633cabc35385410 Jul 30 17:02:14.511708 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.511661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" event={"ID":"c19a42a8-010d-4954-8b4e-af3e11414ce8","Type":"ContainerStarted","Data":"24a1fb986d25312098da030ce6d478232fa12bec056ec69f39661616a2eb89fa"} Jul 30 17:02:14.513197 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.513165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" event={"ID":"ad4fd300-c9e2-4760-902c-40af264af1f3","Type":"ContainerStarted","Data":"7b1c8bbc83a9160ee3addae328073266e4dbfbff43d9ba35f633cabc35385410"} Jul 30 17:02:14.515882 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.515812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsl2b" event={"ID":"02922d64-a0c1-4fc4-91a8-b65655d2e369","Type":"ContainerStarted","Data":"c51e3516c6500e5dd1b97d9a422781caf9564fa37b31dbfd4822e39d74bf5cfc"} Jul 30 17:02:14.515882 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.515858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jsl2b" event={"ID":"02922d64-a0c1-4fc4-91a8-b65655d2e369","Type":"ContainerStarted","Data":"df878a9731f258ce213f8af0af3d7d37c9ba94941a697a708326c5e010cbf482"} Jul 30 17:02:14.562216 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.557825 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jsl2b" podStartSLOduration=3.79096903 podStartE2EDuration="6.557805037s" totalImagesPullingTime="2.659767828s" totalInitContainerRuntime="107.068179ms" isStatefulPod=true podCreationTimestamp="2026-07-30 17:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:14.555678108 +0000 UTC m=+104.228875586" watchObservedRunningTime="2026-07-30 17:02:14.557805037 +0000 UTC m=+104.231002515" Jul 30 17:02:14.675397 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.675342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:14.675931 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.675905 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 17:02:14.694581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.694547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.696518 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.696458 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 17:02:14.697610 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.697581 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 30 17:02:14.697738 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.697606 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 30 17:02:14.697738 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.697668 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 30 17:02:14.697848 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.697597 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 30 17:02:14.697898 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.697876 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 30 17:02:14.697979 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.697962 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 30 17:02:14.698086 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.698070 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t9dxc\"" Jul 30 17:02:14.698465 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.698152 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 30 17:02:14.698575 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.698558 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 30 17:02:14.698764 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.698736 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2qtil24ocqg2e\"" Jul 30 17:02:14.698870 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.698854 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 30 17:02:14.698949 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.698935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 30 17:02:14.699022 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.699003 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 30 17:02:14.699072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.698937 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 30 17:02:14.699520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.699499 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 30 17:02:14.701581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.701560 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 30 17:02:14.704449 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.704429 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 30 17:02:14.802800 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.802710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.802800 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.802765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803035 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.802798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803035 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.802824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-web-config\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803035 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.802882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwtd\" (UniqueName: \"kubernetes.io/projected/d46b64b2-d813-4dbc-84cf-b80ce6f79769-kube-api-access-9lwtd\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803035 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.802909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803035 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b64b2-d813-4dbc-84cf-b80ce6f79769-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803233 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803233 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-config\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803233 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803233 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803404 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803404 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803404 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b64b2-d813-4dbc-84cf-b80ce6f79769-config-out\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803404 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803578 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803578 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803578 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803712 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.803712 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.803668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.904528 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.904746 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.904746 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.904746 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.904746 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.904746 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.904746 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-web-config\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwtd\" (UniqueName: \"kubernetes.io/projected/d46b64b2-d813-4dbc-84cf-b80ce6f79769-kube-api-access-9lwtd\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b64b2-d813-4dbc-84cf-b80ce6f79769-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-config\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.904985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.905020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.905064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.905091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.905122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b64b2-d813-4dbc-84cf-b80ce6f79769-config-out\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.905155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.905469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.905546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.906014 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.906520 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.906060 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.907467 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.906206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.907467 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.906392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.908126 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.908099 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.909020 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.908957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.909692 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.909666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d46b64b2-d813-4dbc-84cf-b80ce6f79769-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.911888 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.911024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.911888 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.911413 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-web-config\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.911888 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.911522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.911888 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.911591 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.911888 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.911663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.911888 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.911853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.912957 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.912937 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b64b2-d813-4dbc-84cf-b80ce6f79769-config-out\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.913238 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.913218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.913309 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.913264 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b64b2-d813-4dbc-84cf-b80ce6f79769-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.914325 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.914296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwtd\" (UniqueName: \"kubernetes.io/projected/d46b64b2-d813-4dbc-84cf-b80ce6f79769-kube-api-access-9lwtd\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:14.914435 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:14.914324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d46b64b2-d813-4dbc-84cf-b80ce6f79769-config\") pod \"prometheus-k8s-0\" (UID: \"d46b64b2-d813-4dbc-84cf-b80ce6f79769\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:15.007876 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:15.007760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:15.895238 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:15.895199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:15.956975 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:15.956928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:16.887698 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:16.887663 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 17:02:16.982076 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:02:16.981997 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46b64b2_d813_4dbc_84cf_b80ce6f79769.slice/crio-b8196f1190dfad1abb9f4db8b2ab8cd5d16d1a5f5f93a7547b49b61cfb408b37 WatchSource:0}: Error finding container b8196f1190dfad1abb9f4db8b2ab8cd5d16d1a5f5f93a7547b49b61cfb408b37: Status 404 returned error can't find the container with id b8196f1190dfad1abb9f4db8b2ab8cd5d16d1a5f5f93a7547b49b61cfb408b37 Jul 30 17:02:17.535475 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.535367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerStarted","Data":"024947a831f223c9cbeba529f1d40cde13380730d1a0174383f92c14951bdc18"} Jul 30 17:02:17.535666 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.535480 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerStarted","Data":"e438bf614c1678ea87f55162af870c89f6b99b4fdd93fdd4179ee9c0e7fd3541"} Jul 30 17:02:17.535666 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.535529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerStarted","Data":"791ef39eb169b6681822ff59794b9b52305bfb13e8d1dcab90ec5d600ead65d7"} Jul 30 17:02:17.535666 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.535568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerStarted","Data":"ce39265a207a988d6b307f286247bfe3fa60296be3b91408a6e1132aa421826f"} Jul 30 17:02:17.535666 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.535602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerStarted","Data":"ec4eab88f6d8f1fe5a71ff6071e92ecf156ce61359f10e902cc33ae09d93eb37"} Jul 30 17:02:17.537158 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.537128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" event={"ID":"c19a42a8-010d-4954-8b4e-af3e11414ce8","Type":"ContainerStarted","Data":"e6a0483286e4ba23a7fb2a36a294f87b2bed2731b70654d472d175d62f2c489f"} Jul 30 17:02:17.537357 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.537340 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:17.538702 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.538674 2576 generic.go:356] "Generic (PLEG): container finished" podID="d46b64b2-d813-4dbc-84cf-b80ce6f79769" containerID="7741464e6ffa8005691078d3038ee18e23796b949685811f5b69c88de35a5f9d" exitCode=0 Jul 30 17:02:17.538823 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.538763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerDied","Data":"7741464e6ffa8005691078d3038ee18e23796b949685811f5b69c88de35a5f9d"} Jul 30 17:02:17.538884 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.538833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerStarted","Data":"b8196f1190dfad1abb9f4db8b2ab8cd5d16d1a5f5f93a7547b49b61cfb408b37"} Jul 30 17:02:17.541070 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.541050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" event={"ID":"ad4fd300-c9e2-4760-902c-40af264af1f3","Type":"ContainerStarted","Data":"a139ca4e140dd93ae51694987f4f4db7be2a145e4c189d1e4032749ea53a422a"} Jul 30 17:02:17.541163 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.541088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" event={"ID":"ad4fd300-c9e2-4760-902c-40af264af1f3","Type":"ContainerStarted","Data":"2065e07f0ccb042a4692a14a1b55928066df95a3a0c399e6f3c4f3a567dbf579"} Jul 30 17:02:17.541163 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.541108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" event={"ID":"ad4fd300-c9e2-4760-902c-40af264af1f3","Type":"ContainerStarted","Data":"6b377c96168b4905f781fe8820dd94e167916fb7296fde687b06816e766cad65"} Jul 30 17:02:17.543078 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.543058 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" Jul 30 17:02:17.554459 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.554415 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jwfzd" podStartSLOduration=1.462879582 podStartE2EDuration="4.554402245s" totalImagesPullingTime="3.091522663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:17.553632454 +0000 UTC m=+107.226829932" watchObservedRunningTime="2026-07-30 17:02:17.554402245 +0000 UTC m=+107.227599716" Jul 30 17:02:17.575877 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:17.575823 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-555c98fb9b-6w9nj" podStartSLOduration=1.6419375939999998 podStartE2EDuration="4.575805533s" totalImagesPullingTime="2.933867939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:17.575005439 +0000 UTC m=+107.248202915" watchObservedRunningTime="2026-07-30 17:02:17.575805533 +0000 UTC m=+107.249003011" Jul 30 17:02:18.989258 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:18.989218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:19.080836 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:19.080750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:19.181348 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:19.181306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:19.247893 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:19.247848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:19.554640 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:19.554606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fc407a1-e51b-4652-a677-935c67c6a364","Type":"ContainerStarted","Data":"eed4d6d6b73727ba55d733d0bf99e445e5c77f19a832c6d4accc4e28475f22bd"} Jul 30 17:02:19.581057 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:19.581001 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.663718295 podStartE2EDuration="10.580984248s" totalImagesPullingTime="6.89245724s" totalInitContainerRuntime="24.808713ms" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:19.579685797 +0000 UTC m=+109.252883276" watchObservedRunningTime="2026-07-30 17:02:19.580984248 +0000 UTC m=+109.254181725" Jul 30 17:02:19.699569 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:19.699538 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:19.802397 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:19.802291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:21.567970 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:21.567937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerStarted","Data":"a7367b854cb71ff41adbea9bf9c0fbadf9d2256b7f529508b65ce1cf98c7596d"} Jul 30 17:02:21.568415 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:21.568045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerStarted","Data":"794927ba8110e13e4b86e59b0dc0220549141c75e53b0336cb529a526b6a93b4"} Jul 30 17:02:23.565589 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:23.565563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:23.670007 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:23.669958 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:23.768778 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:23.768721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:24.585388 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:24.585350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerStarted","Data":"85452a838d61bb40ec56ef9a73f5fc1cb3eb0f52f4c18060ccf45f606524e26f"} Jul 30 17:02:24.585846 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:24.585452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerStarted","Data":"52bf47e11bfa751c3d4058fd16b7cba94165d29d7d6d1fa2c4efba3ae732b4c0"} Jul 30 17:02:24.585846 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:24.585498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerStarted","Data":"890d37c875dee17b42dcf7e4f478418d54b738ae408c9ac33d22c18596e06fb5"} Jul 30 17:02:24.585846 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:24.585532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d46b64b2-d813-4dbc-84cf-b80ce6f79769","Type":"ContainerStarted","Data":"f2d9f237e8f4607ba86748266fa03a8b91db0556f6c3883424ab7cbed4d5847b"} Jul 30 17:02:24.616028 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:24.615980 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.862142833 podStartE2EDuration="10.615963754s" totalImagesPullingTime="5.722115052s" totalInitContainerRuntime="31.705869ms" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:24.613531299 +0000 UTC m=+114.286728776" watchObservedRunningTime="2026-07-30 17:02:24.615963754 +0000 UTC m=+114.289161230" Jul 30 17:02:24.968834 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:24.968801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:25.008415 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:25.008359 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:02:25.031021 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:25.030983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:25.086889 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:25.086853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:25.968352 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:25.968317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:26.040522 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:26.040489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:29.924781 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:29.924744 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cfc4576cf-kcfv8"] Jul 30 17:02:30.034429 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:30.034394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:30.135138 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:30.135108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:30.235252 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:30.235220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:30.292959 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:30.292924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:02:32.547218 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.547159 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7696fdb7bb-nftqx" podUID="6f1598f9-81bd-4742-9520-6c51417cc888" containerName="console" containerID="cri-o://49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3" gracePeriod=15 Jul 30 17:02:32.822558 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.822495 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:02:32.978343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978301 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-oauth-serving-cert\") pod \"6f1598f9-81bd-4742-9520-6c51417cc888\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " Jul 30 17:02:32.978343 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978341 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-service-ca\") pod \"6f1598f9-81bd-4742-9520-6c51417cc888\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " Jul 30 17:02:32.978604 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-serving-cert\") pod \"6f1598f9-81bd-4742-9520-6c51417cc888\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " Jul 30 17:02:32.978604 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978458 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-oauth-config\") pod \"6f1598f9-81bd-4742-9520-6c51417cc888\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " Jul 30 17:02:32.978604 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74p8r\" (UniqueName: \"kubernetes.io/projected/6f1598f9-81bd-4742-9520-6c51417cc888-kube-api-access-74p8r\") pod \"6f1598f9-81bd-4742-9520-6c51417cc888\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " Jul 30 17:02:32.978604 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978560 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-console-config\") pod \"6f1598f9-81bd-4742-9520-6c51417cc888\" (UID: \"6f1598f9-81bd-4742-9520-6c51417cc888\") " Jul 30 17:02:32.978859 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978806 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6f1598f9-81bd-4742-9520-6c51417cc888" (UID: "6f1598f9-81bd-4742-9520-6c51417cc888"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:02:32.978941 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978911 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-service-ca" (OuterVolumeSpecName: "service-ca") pod "6f1598f9-81bd-4742-9520-6c51417cc888" (UID: "6f1598f9-81bd-4742-9520-6c51417cc888"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:02:32.978991 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.978969 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-console-config" (OuterVolumeSpecName: "console-config") pod "6f1598f9-81bd-4742-9520-6c51417cc888" (UID: "6f1598f9-81bd-4742-9520-6c51417cc888"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:02:32.980776 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.980747 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6f1598f9-81bd-4742-9520-6c51417cc888" (UID: "6f1598f9-81bd-4742-9520-6c51417cc888"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:02:32.980776 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.980765 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6f1598f9-81bd-4742-9520-6c51417cc888" (UID: "6f1598f9-81bd-4742-9520-6c51417cc888"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:02:32.980945 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:32.980876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1598f9-81bd-4742-9520-6c51417cc888-kube-api-access-74p8r" (OuterVolumeSpecName: "kube-api-access-74p8r") pod "6f1598f9-81bd-4742-9520-6c51417cc888" (UID: "6f1598f9-81bd-4742-9520-6c51417cc888"). InnerVolumeSpecName "kube-api-access-74p8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:02:33.080011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.079934 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-serving-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:33.080011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.079955 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f1598f9-81bd-4742-9520-6c51417cc888-console-oauth-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:33.080011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.079966 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74p8r\" (UniqueName: \"kubernetes.io/projected/6f1598f9-81bd-4742-9520-6c51417cc888-kube-api-access-74p8r\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:33.080011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.079977 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-console-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:33.080011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.079985 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-oauth-serving-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:33.080011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.079994 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1598f9-81bd-4742-9520-6c51417cc888-service-ca\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:33.632445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.632409 2576 generic.go:356] "Generic (PLEG): container finished" podID="6f1598f9-81bd-4742-9520-6c51417cc888" containerID="49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3" exitCode=0 Jul 30 17:02:33.632840 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.632459 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7696fdb7bb-nftqx" Jul 30 17:02:33.632840 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.632498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7696fdb7bb-nftqx" event={"ID":"6f1598f9-81bd-4742-9520-6c51417cc888","Type":"ContainerDied","Data":"49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3"} Jul 30 17:02:33.632840 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.632551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7696fdb7bb-nftqx" event={"ID":"6f1598f9-81bd-4742-9520-6c51417cc888","Type":"ContainerDied","Data":"d4a8613d47ce521c9c715ed95517121d05b73dd17e74b812c6bcdc4b0721d2d5"} Jul 30 17:02:33.632840 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.632578 2576 scope.go:117] "RemoveContainer" containerID="49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3" Jul 30 17:02:33.643795 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.643778 2576 scope.go:117] "RemoveContainer" containerID="49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3" Jul 30 17:02:33.644060 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:33.644035 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3\": container with ID starting with 49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3 not found: ID does not exist" containerID="49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3" Jul 30 17:02:33.644151 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.644062 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3"} err="failed to get container status \"49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3\": rpc error: code = NotFound desc = could not find container \"49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3\": container with ID starting with 49e2cfa1588298732eeec099c40ea723931ab8edf4609c048bd4beb1f3c061f3 not found: ID does not exist" Jul 30 17:02:33.655611 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.655584 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7696fdb7bb-nftqx"] Jul 30 17:02:33.660623 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:33.660604 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7696fdb7bb-nftqx"] Jul 30 17:02:34.938660 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:34.938617 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f1598f9-81bd-4742-9520-6c51417cc888" path="/var/lib/kubelet/pods/6f1598f9-81bd-4742-9520-6c51417cc888/volumes" Jul 30 17:02:36.647281 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:36.647229 2576 generic.go:356] "Generic (PLEG): container finished" podID="45bb5507-77d6-4be7-bc89-95aa2d5f02b6" containerID="196c111707c53ea73ef8e0e95370e155dad0cf731c033baabac651c76fcf9967" exitCode=0 Jul 30 17:02:36.647695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:36.647302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" event={"ID":"45bb5507-77d6-4be7-bc89-95aa2d5f02b6","Type":"ContainerDied","Data":"196c111707c53ea73ef8e0e95370e155dad0cf731c033baabac651c76fcf9967"} Jul 30 17:02:36.647695 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:36.647646 2576 scope.go:117] "RemoveContainer" containerID="196c111707c53ea73ef8e0e95370e155dad0cf731c033baabac651c76fcf9967" Jul 30 17:02:37.653200 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:37.653161 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-p2jdk" event={"ID":"45bb5507-77d6-4be7-bc89-95aa2d5f02b6","Type":"ContainerStarted","Data":"b47c62a87e2b50a907b7ca953273b969af14610c851ea3505207de8f61a11e3c"} Jul 30 17:02:47.695512 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:47.695475 2576 generic.go:356] "Generic (PLEG): container finished" podID="2633ff8f-337d-48db-ace0-2316e62d08b5" containerID="814881cb60b4fbbc62191af7ae882abe35658bea409e49afacd8a0872373abd9" exitCode=0 Jul 30 17:02:47.695955 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:47.695552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" event={"ID":"2633ff8f-337d-48db-ace0-2316e62d08b5","Type":"ContainerDied","Data":"814881cb60b4fbbc62191af7ae882abe35658bea409e49afacd8a0872373abd9"} Jul 30 17:02:47.696033 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:47.696010 2576 scope.go:117] "RemoveContainer" containerID="814881cb60b4fbbc62191af7ae882abe35658bea409e49afacd8a0872373abd9" Jul 30 17:02:47.697190 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:47.697167 2576 generic.go:356] "Generic (PLEG): container finished" podID="3ec7ff92-af94-4d0f-ace3-5d7adef84338" containerID="e13f74dad5f4f324f3fdeb110ef4c2354ae7f1f2ec3751d3af399ec6d264c00c" exitCode=0 Jul 30 17:02:47.697269 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:47.697238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c795x" event={"ID":"3ec7ff92-af94-4d0f-ace3-5d7adef84338","Type":"ContainerDied","Data":"e13f74dad5f4f324f3fdeb110ef4c2354ae7f1f2ec3751d3af399ec6d264c00c"} Jul 30 17:02:47.697549 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:47.697537 2576 scope.go:117] "RemoveContainer" containerID="e13f74dad5f4f324f3fdeb110ef4c2354ae7f1f2ec3751d3af399ec6d264c00c" Jul 30 17:02:48.702937 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:48.702898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl" event={"ID":"2633ff8f-337d-48db-ace0-2316e62d08b5","Type":"ContainerStarted","Data":"3ec8b4ce11ed6ba5374922f3b86050590dc788b3e97f3ae3d36738b690001be8"} Jul 30 17:02:48.704785 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:48.704760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c795x" event={"ID":"3ec7ff92-af94-4d0f-ace3-5d7adef84338","Type":"ContainerStarted","Data":"66b47e6429782a3ec529252264bd51a2a429a15acac547651216b342ce796fcf"} Jul 30 17:02:54.950015 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:54.949980 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cfc4576cf-kcfv8" podUID="a9d414bf-7d12-4886-ab57-d96d92b4d9e6" containerName="console" containerID="cri-o://3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8" gracePeriod=15 Jul 30 17:02:55.249527 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.249502 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:02:55.360061 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-service-ca\") pod \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " Jul 30 17:02:55.360211 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-config\") pod \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " Jul 30 17:02:55.360211 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360122 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-trusted-ca-bundle\") pod \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " Jul 30 17:02:55.360211 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360169 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-oauth-config\") pod \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " Jul 30 17:02:55.360325 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360235 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-oauth-serving-cert\") pod \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " Jul 30 17:02:55.360325 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360260 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-serving-cert\") pod \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " Jul 30 17:02:55.360325 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360295 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czckg\" (UniqueName: \"kubernetes.io/projected/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-kube-api-access-czckg\") pod \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\" (UID: \"a9d414bf-7d12-4886-ab57-d96d92b4d9e6\") " Jul 30 17:02:55.360500 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360482 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-service-ca" (OuterVolumeSpecName: "service-ca") pod "a9d414bf-7d12-4886-ab57-d96d92b4d9e6" (UID: "a9d414bf-7d12-4886-ab57-d96d92b4d9e6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:02:55.360864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360500 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-config" (OuterVolumeSpecName: "console-config") pod "a9d414bf-7d12-4886-ab57-d96d92b4d9e6" (UID: "a9d414bf-7d12-4886-ab57-d96d92b4d9e6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:02:55.360864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360635 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a9d414bf-7d12-4886-ab57-d96d92b4d9e6" (UID: "a9d414bf-7d12-4886-ab57-d96d92b4d9e6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:02:55.360864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360695 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a9d414bf-7d12-4886-ab57-d96d92b4d9e6" (UID: "a9d414bf-7d12-4886-ab57-d96d92b4d9e6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:02:55.360864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360768 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:55.360864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360787 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-trusted-ca-bundle\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:55.360864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360802 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-oauth-serving-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:55.360864 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.360816 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-service-ca\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:55.362488 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.362462 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-kube-api-access-czckg" (OuterVolumeSpecName: "kube-api-access-czckg") pod "a9d414bf-7d12-4886-ab57-d96d92b4d9e6" (UID: "a9d414bf-7d12-4886-ab57-d96d92b4d9e6"). InnerVolumeSpecName "kube-api-access-czckg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:02:55.362567 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.362541 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a9d414bf-7d12-4886-ab57-d96d92b4d9e6" (UID: "a9d414bf-7d12-4886-ab57-d96d92b4d9e6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:02:55.362612 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.362600 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a9d414bf-7d12-4886-ab57-d96d92b4d9e6" (UID: "a9d414bf-7d12-4886-ab57-d96d92b4d9e6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:02:55.462158 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.462089 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-oauth-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:55.462158 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.462116 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-console-serving-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:55.462158 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.462125 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czckg\" (UniqueName: \"kubernetes.io/projected/a9d414bf-7d12-4886-ab57-d96d92b4d9e6-kube-api-access-czckg\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:02:55.739229 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.739129 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9d414bf-7d12-4886-ab57-d96d92b4d9e6" containerID="3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8" exitCode=0 Jul 30 17:02:55.739229 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.739198 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cfc4576cf-kcfv8" Jul 30 17:02:55.739229 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.739213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfc4576cf-kcfv8" event={"ID":"a9d414bf-7d12-4886-ab57-d96d92b4d9e6","Type":"ContainerDied","Data":"3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8"} Jul 30 17:02:55.739529 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.739269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cfc4576cf-kcfv8" event={"ID":"a9d414bf-7d12-4886-ab57-d96d92b4d9e6","Type":"ContainerDied","Data":"6477ebc255259055817daa079c8958f49b5daee1ab1c9049a487cb0d90b2014c"} Jul 30 17:02:55.739529 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.739299 2576 scope.go:117] "RemoveContainer" containerID="3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8" Jul 30 17:02:55.759416 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.759399 2576 scope.go:117] "RemoveContainer" containerID="3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8" Jul 30 17:02:55.759667 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:02:55.759650 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8\": container with ID starting with 3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8 not found: ID does not exist" containerID="3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8" Jul 30 17:02:55.759716 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.759683 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8"} err="failed to get container status \"3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8\": rpc error: code = NotFound desc = could not find container \"3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8\": container with ID starting with 3878bf88da3ac1d07a222c18ed57a8188071252dd43e65a91606f63ef93f94f8 not found: ID does not exist" Jul 30 17:02:55.761893 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.761871 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cfc4576cf-kcfv8"] Jul 30 17:02:55.765708 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:55.765688 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cfc4576cf-kcfv8"] Jul 30 17:02:56.938503 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:02:56.938469 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9d414bf-7d12-4886-ab57-d96d92b4d9e6" path="/var/lib/kubelet/pods/a9d414bf-7d12-4886-ab57-d96d92b4d9e6/volumes" Jul 30 17:03:15.008418 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:15.008361 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:03:15.027946 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:15.027918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:03:15.851279 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:15.851249 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 17:03:24.460224 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:24.460130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:24.514238 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:24.514205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:31.141306 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:31.141214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:31.202573 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:31.202543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:31.267942 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:31.267901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:31.321568 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:31.321537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:31.865934 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:31.865896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:31.920613 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:31.920570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:36.417200 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:36.417162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:36.518492 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:36.518454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:38.809260 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:38.809220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:42.490838 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:42.490808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:42.545632 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:42.545599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:42.600959 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:42.600924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:53.581363 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:53.581327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:53.681212 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:53.681186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:03:55.002232 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:03:55.002179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:04:01.627119 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:04:01.627084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:04:01.728584 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:04:01.728542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:04:02.551125 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:04:02.551088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:04:18.887018 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:04:18.886976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:04:31.945065 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:04:31.945014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:05:30.747835 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:05:30.747808 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 17:05:37.380901 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:05:37.380862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:05:38.474768 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:05:38.474728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:05:59.548840 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:05:59.548804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:05:59.628636 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:05:59.628603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:06.517569 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:06.517521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:06.849579 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:06.849482 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:07.222995 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:07.222957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:07.585087 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:07.584990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:07.700197 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:07.700153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:07.854212 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:07.854116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:07.965531 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:07.965506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:08.062915 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:08.062879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:09.186007 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:09.185976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:09.292367 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:09.292336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:59.259138 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:59.259103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:06:59.318446 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:06:59.318418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:03.716781 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:03.716752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:20.681830 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:20.681793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:20.782017 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:20.781987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:20.881745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:20.881722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:21.390139 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:21.390106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:21.450138 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:21.450105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.028865 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.028811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.126423 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.126337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.186684 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.186656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.285072 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.285002 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.386268 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.386240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.604035 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.603955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.704159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.704101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:27.758844 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:27.758805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:31.455797 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:31.455765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:31.556927 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:31.556893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:07:57.625701 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:07:57.625611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:03.471120 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:03.471081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:44.046087 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:44.046039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:44.126987 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:44.126950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:44.263045 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:44.263011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:44.362073 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:44.361987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:44.460161 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:44.460136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:44.754850 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:44.754812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:08:44.807946 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:08:44.807915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:09:00.027175 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.027145 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-nvnpg"] Jul 30 17:09:00.027584 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.027507 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1598f9-81bd-4742-9520-6c51417cc888" containerName="console" Jul 30 17:09:00.027584 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.027519 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1598f9-81bd-4742-9520-6c51417cc888" containerName="console" Jul 30 17:09:00.027584 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.027531 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d414bf-7d12-4886-ab57-d96d92b4d9e6" containerName="console" Jul 30 17:09:00.027584 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.027536 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d414bf-7d12-4886-ab57-d96d92b4d9e6" containerName="console" Jul 30 17:09:00.027724 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.027600 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d414bf-7d12-4886-ab57-d96d92b4d9e6" containerName="console" Jul 30 17:09:00.027724 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.027608 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1598f9-81bd-4742-9520-6c51417cc888" containerName="console" Jul 30 17:09:00.029685 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.029670 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.032681 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.032660 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 30 17:09:00.032792 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.032663 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 30 17:09:00.033540 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.033524 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2z58f\"" Jul 30 17:09:00.033639 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.033548 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 30 17:09:00.038318 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.038291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nvnpg"] Jul 30 17:09:00.113301 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.113268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-model-cache\") pod \"s3-init-nvnpg\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.113478 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.113319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwl4\" (UniqueName: \"kubernetes.io/projected/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-kube-api-access-kbwl4\") pod \"s3-init-nvnpg\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.213875 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.213839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-model-cache\") pod \"s3-init-nvnpg\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.214016 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.213883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwl4\" (UniqueName: \"kubernetes.io/projected/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-kube-api-access-kbwl4\") pod \"s3-init-nvnpg\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.214221 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.214201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-model-cache\") pod \"s3-init-nvnpg\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.223697 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.223666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwl4\" (UniqueName: \"kubernetes.io/projected/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-kube-api-access-kbwl4\") pod \"s3-init-nvnpg\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.343519 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.343421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nvnpg" Jul 30 17:09:00.477192 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.477162 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nvnpg"] Jul 30 17:09:00.477720 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:09:00.477693 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b147fe4_9a73_46a3_b456_bd68f9e8a43c.slice/crio-678484200e98cadb0f18121be1dcd13c67bf084a3d5ab844eac252bfe86f8112 WatchSource:0}: Error finding container 678484200e98cadb0f18121be1dcd13c67bf084a3d5ab844eac252bfe86f8112: Status 404 returned error can't find the container with id 678484200e98cadb0f18121be1dcd13c67bf084a3d5ab844eac252bfe86f8112 Jul 30 17:09:00.479651 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:00.479631 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 17:09:01.198224 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:01.198184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nvnpg" event={"ID":"6b147fe4-9a73-46a3-b456-bd68f9e8a43c","Type":"ContainerStarted","Data":"678484200e98cadb0f18121be1dcd13c67bf084a3d5ab844eac252bfe86f8112"} Jul 30 17:09:04.213052 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:04.213012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nvnpg" event={"ID":"6b147fe4-9a73-46a3-b456-bd68f9e8a43c","Type":"ContainerStarted","Data":"ade58023ed3a174d78e5604bf868add58d506722c6abbe833fc5e8d9a0909225"} Jul 30 17:09:06.222851 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:06.222816 2576 generic.go:356] "Generic (PLEG): container finished" podID="6b147fe4-9a73-46a3-b456-bd68f9e8a43c" containerID="ade58023ed3a174d78e5604bf868add58d506722c6abbe833fc5e8d9a0909225" exitCode=0 Jul 30 17:09:06.223210 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:06.222891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nvnpg" event={"ID":"6b147fe4-9a73-46a3-b456-bd68f9e8a43c","Type":"ContainerDied","Data":"ade58023ed3a174d78e5604bf868add58d506722c6abbe833fc5e8d9a0909225"} Jul 30 17:09:11.247007 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:11.246975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nvnpg" event={"ID":"6b147fe4-9a73-46a3-b456-bd68f9e8a43c","Type":"ContainerStarted","Data":"b8ba11fea937bbad07520b577d55df4a452c0ead6a512ee093328f24c473030b"} Jul 30 17:09:11.265191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:11.265146 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-nvnpg" podStartSLOduration=1.370808584 podStartE2EDuration="11.265132455s" totalImagesPullingTime="7.726542175s" totalInitContainerRuntime="2.167781696s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:09:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:09:11.264919703 +0000 UTC m=+520.938117181" watchObservedRunningTime="2026-07-30 17:09:11.265132455 +0000 UTC m=+520.938329943" Jul 30 17:09:13.151634 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:13.151600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:09:34.467008 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:34.466921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:09:34.519256 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:09:34.519219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:10:07.757270 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:07.757236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:10:16.508807 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:16.508771 2576 generic.go:356] "Generic (PLEG): container finished" podID="6b147fe4-9a73-46a3-b456-bd68f9e8a43c" containerID="b8ba11fea937bbad07520b577d55df4a452c0ead6a512ee093328f24c473030b" exitCode=0 Jul 30 17:10:16.509198 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:16.508836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nvnpg" event={"ID":"6b147fe4-9a73-46a3-b456-bd68f9e8a43c","Type":"ContainerDied","Data":"b8ba11fea937bbad07520b577d55df4a452c0ead6a512ee093328f24c473030b"} Jul 30 17:10:17.643877 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:17.643850 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nvnpg" Jul 30 17:10:17.781646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:17.781559 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbwl4\" (UniqueName: \"kubernetes.io/projected/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-kube-api-access-kbwl4\") pod \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " Jul 30 17:10:17.781646 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:17.781635 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-model-cache\") pod \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\" (UID: \"6b147fe4-9a73-46a3-b456-bd68f9e8a43c\") " Jul 30 17:10:17.783777 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:17.783752 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-kube-api-access-kbwl4" (OuterVolumeSpecName: "kube-api-access-kbwl4") pod "6b147fe4-9a73-46a3-b456-bd68f9e8a43c" (UID: "6b147fe4-9a73-46a3-b456-bd68f9e8a43c"). InnerVolumeSpecName "kube-api-access-kbwl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:10:17.800159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:17.800133 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b147fe4-9a73-46a3-b456-bd68f9e8a43c" (UID: "6b147fe4-9a73-46a3-b456-bd68f9e8a43c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 17:10:17.882486 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:17.882455 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-model-cache\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:10:17.882486 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:17.882480 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbwl4\" (UniqueName: \"kubernetes.io/projected/6b147fe4-9a73-46a3-b456-bd68f9e8a43c-kube-api-access-kbwl4\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:10:18.521219 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:18.521191 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nvnpg" Jul 30 17:10:18.521412 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:18.521194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nvnpg" event={"ID":"6b147fe4-9a73-46a3-b456-bd68f9e8a43c","Type":"ContainerDied","Data":"678484200e98cadb0f18121be1dcd13c67bf084a3d5ab844eac252bfe86f8112"} Jul 30 17:10:18.521412 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:18.521302 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="678484200e98cadb0f18121be1dcd13c67bf084a3d5ab844eac252bfe86f8112" Jul 30 17:10:18.879253 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:18.879153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:10:18.980676 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:18.980648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:10:28.658803 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:28.658765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:10:30.466844 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:30.466807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:10:30.738650 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:10:30.738562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:11:50.896635 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:11:50.896580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:12:25.683760 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:12:25.683642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:12:25.749980 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:12:25.749946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:12:25.937652 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:12:25.937560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:12:26.042893 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:12:26.042859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:12:26.138852 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:12:26.138822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:15.791357 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:15.791308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:16.465211 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:16.465161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:25.833832 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:25.833801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:25.988780 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:25.988738 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:26.060104 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:26.060072 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:26.164208 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:26.164128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:26.246331 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:26.246298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:26.346997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:26.346968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:14:26.594095 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:14:26.594060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:16.121617 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:16.121581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:16.369055 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:16.369025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:25.190452 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:25.190421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:25.291898 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:25.291859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:26.158186 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:26.158159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:26.386290 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:26.386238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:26.499776 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:26.499749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:26.917944 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:26.917850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:46.338607 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:46.338567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:16:46.754254 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:16:46.754199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:18:31.665751 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:18:31.665669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:18:33.116084 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:18:33.116046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:19:43.985056 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:19:43.985024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:20:08.268137 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:20:08.268058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:21:31.884390 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:21:31.884296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:21:36.952245 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:21:36.952207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:21:37.011757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:21:37.011727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:21:37.116708 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:21:37.116672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:21:37.217420 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:21:37.217305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:22:47.586515 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:22:47.586480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:22:47.640853 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:22:47.640819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:23:07.125144 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:23:07.125054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:23:07.186486 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:23:07.186451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:23:12.206847 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:23:12.206807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:23:12.307237 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:23:12.307202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:23:12.389907 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:23:12.389881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:23:12.490191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:23:12.490113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:24:52.433746 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.433714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:24:52.480505 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.480473 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472"] Jul 30 17:24:52.480799 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.480786 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b147fe4-9a73-46a3-b456-bd68f9e8a43c" containerName="download-hf-model" Jul 30 17:24:52.480850 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.480800 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b147fe4-9a73-46a3-b456-bd68f9e8a43c" containerName="download-hf-model" Jul 30 17:24:52.480850 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.480816 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b147fe4-9a73-46a3-b456-bd68f9e8a43c" containerName="s3-init" Jul 30 17:24:52.480850 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.480822 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b147fe4-9a73-46a3-b456-bd68f9e8a43c" containerName="s3-init" Jul 30 17:24:52.480951 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.480882 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b147fe4-9a73-46a3-b456-bd68f9e8a43c" containerName="s3-init" Jul 30 17:24:52.486436 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.486411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.490177 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.490158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qz9j4\"" Jul 30 17:24:52.490430 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.490414 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 30 17:24:52.490544 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.490525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Jul 30 17:24:52.490611 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.490531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 30 17:24:52.490611 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.490531 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Jul 30 17:24:52.490829 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.490811 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 30 17:24:52.491645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.491631 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 30 17:24:52.496528 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.496509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472"] Jul 30 17:24:52.572442 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.572423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.572552 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.572448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.572552 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.572468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.572633 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.572561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz58w\" (UniqueName: \"kubernetes.io/projected/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-api-access-xz58w\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.572633 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.572611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.673370 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.673342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz58w\" (UniqueName: \"kubernetes.io/projected/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-api-access-xz58w\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.673485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.673403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.673485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.673440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.673485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.673455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.673485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.673474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.673897 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.673881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.674216 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.674199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.674260 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.674207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.675982 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.675962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.683617 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.683592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz58w\" (UniqueName: \"kubernetes.io/projected/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-api-access-xz58w\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-ts472\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.797562 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.797533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:24:52.925448 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:52.925422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472"] Jul 30 17:24:52.927410 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:24:52.927362 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0c6d2d_85f3_4d03_b8aa_738f435c5a6c.slice/crio-252cb3bd29826a7eab00928ed70f934abb112b2ca25af37a18b413a1e8716997 WatchSource:0}: Error finding container 252cb3bd29826a7eab00928ed70f934abb112b2ca25af37a18b413a1e8716997: Status 404 returned error can't find the container with id 252cb3bd29826a7eab00928ed70f934abb112b2ca25af37a18b413a1e8716997 Jul 30 17:24:53.003186 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:53.003155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerStarted","Data":"12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a"} Jul 30 17:24:53.003277 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:53.003222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerStarted","Data":"252cb3bd29826a7eab00928ed70f934abb112b2ca25af37a18b413a1e8716997"} Jul 30 17:24:57.024070 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:57.024040 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerID="12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a" exitCode=0 Jul 30 17:24:57.024464 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:57.024118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerDied","Data":"12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a"} Jul 30 17:24:57.025194 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:24:57.025177 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 17:25:12.102859 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:12.102817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerStarted","Data":"c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e"} Jul 30 17:25:14.114325 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:14.114287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerStarted","Data":"ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73"} Jul 30 17:25:14.114726 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:14.114471 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:25:14.153267 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:14.153212 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podStartSLOduration=1.782046094 podStartE2EDuration="22.153191349s" totalImagesPullingTime="16.504804636s" totalInitContainerRuntime="3.866340619s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:24:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:25:14.152749587 +0000 UTC m=+1483.825947064" watchObservedRunningTime="2026-07-30 17:25:14.153191349 +0000 UTC m=+1483.826388830" Jul 30 17:25:15.118989 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:15.118961 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:25:15.120289 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:15.120258 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:25:16.123130 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:16.123093 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:25:21.128599 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:21.128566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:25:21.129295 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:21.129260 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:25:31.130063 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:31.130023 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:25:41.129673 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:41.129633 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:25:51.129179 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:25:51.129139 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:26:01.129526 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:01.129488 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:26:11.129641 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:11.129603 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:26:21.129685 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:21.129648 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Jul 30 17:26:31.130080 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:31.130053 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:26:32.870052 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:32.870020 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472"] Jul 30 17:26:32.871573 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:32.870263 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" containerID="cri-o://c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e" gracePeriod=30 Jul 30 17:26:32.871573 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:32.870275 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kube-rbac-proxy" containerID="cri-o://ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73" gracePeriod=30 Jul 30 17:26:33.446230 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:33.446197 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerID="ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73" exitCode=2 Jul 30 17:26:33.446428 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:33.446277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerDied","Data":"ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73"} Jul 30 17:26:33.900944 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:33.900898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:26:36.123832 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:36.123788 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused" Jul 30 17:26:37.618427 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.618402 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:26:37.700939 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.700901 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-rbac-proxy-sar-config\") pod \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " Jul 30 17:26:37.700939 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.700946 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz58w\" (UniqueName: \"kubernetes.io/projected/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-api-access-xz58w\") pod \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " Jul 30 17:26:37.701213 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.700988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kserve-provision-location\") pod \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " Jul 30 17:26:37.701213 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.701011 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-cabundle-cert\") pod \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " Jul 30 17:26:37.701213 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.701054 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-proxy-tls\") pod \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\" (UID: \"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c\") " Jul 30 17:26:37.701412 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.701330 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" (UID: "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 17:26:37.701412 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.701347 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" (UID: "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:26:37.701516 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.701452 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" (UID: "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:26:37.703203 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.703182 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" (UID: "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:26:37.703286 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.703268 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-api-access-xz58w" (OuterVolumeSpecName: "kube-api-access-xz58w") pod "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" (UID: "6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c"). InnerVolumeSpecName "kube-api-access-xz58w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:26:37.802440 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.802403 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kserve-provision-location\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:26:37.802440 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.802438 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-cabundle-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:26:37.802640 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.802453 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-proxy-tls\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:26:37.802640 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.802466 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:26:37.802640 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.802480 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz58w\" (UniqueName: \"kubernetes.io/projected/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c-kube-api-access-xz58w\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:26:37.809900 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.809871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:26:37.910460 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.910424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:26:37.972773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.972688 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd"] Jul 30 17:26:37.973015 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973003 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" Jul 30 17:26:37.973060 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973017 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" Jul 30 17:26:37.973060 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973036 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kube-rbac-proxy" Jul 30 17:26:37.973060 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973041 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kube-rbac-proxy" Jul 30 17:26:37.973060 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973058 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="storage-initializer" Jul 30 17:26:37.973177 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973066 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="storage-initializer" Jul 30 17:26:37.973177 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973117 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kserve-container" Jul 30 17:26:37.973177 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.973123 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerName="kube-rbac-proxy" Jul 30 17:26:37.975163 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.975147 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:37.979745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.979728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Jul 30 17:26:37.980040 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.980023 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Jul 30 17:26:37.999816 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:37.999787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd"] Jul 30 17:26:38.004295 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.004270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhw6\" (UniqueName: \"kubernetes.io/projected/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-api-access-zmhw6\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.004433 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.004308 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.004433 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.004338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.004433 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.004420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.004588 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.004482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.013813 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.013788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:26:38.104902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.104868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhw6\" (UniqueName: \"kubernetes.io/projected/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-api-access-zmhw6\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.105094 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.104915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.105094 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.104936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.105094 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.105054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.105254 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.105168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.105354 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.105336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.105773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.105750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.105847 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.105794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.107384 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.107356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.125223 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.125200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhw6\" (UniqueName: \"kubernetes.io/projected/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-api-access-zmhw6\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.134837 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.134814 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:26:38.287216 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.287125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:38.420326 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.420304 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd"] Jul 30 17:26:38.421957 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:26:38.421920 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a8b32a_4eea_44f3_95dc_dfbdc6f61e0d.slice/crio-8f0171ebb72a889a7543e5f098dfe1ea02ae3a2bf0fc28b1097d06469a6aad36 WatchSource:0}: Error finding container 8f0171ebb72a889a7543e5f098dfe1ea02ae3a2bf0fc28b1097d06469a6aad36: Status 404 returned error can't find the container with id 8f0171ebb72a889a7543e5f098dfe1ea02ae3a2bf0fc28b1097d06469a6aad36 Jul 30 17:26:38.469704 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.469676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerStarted","Data":"8f0171ebb72a889a7543e5f098dfe1ea02ae3a2bf0fc28b1097d06469a6aad36"} Jul 30 17:26:38.471297 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.471267 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" containerID="c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e" exitCode=0 Jul 30 17:26:38.471401 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.471324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerDied","Data":"c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e"} Jul 30 17:26:38.471401 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.471389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" event={"ID":"6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c","Type":"ContainerDied","Data":"252cb3bd29826a7eab00928ed70f934abb112b2ca25af37a18b413a1e8716997"} Jul 30 17:26:38.471478 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.471332 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472" Jul 30 17:26:38.471478 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.471424 2576 scope.go:117] "RemoveContainer" containerID="ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73" Jul 30 17:26:38.480081 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.480062 2576 scope.go:117] "RemoveContainer" containerID="c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e" Jul 30 17:26:38.488848 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.488831 2576 scope.go:117] "RemoveContainer" containerID="12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a" Jul 30 17:26:38.497191 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.497169 2576 scope.go:117] "RemoveContainer" containerID="ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73" Jul 30 17:26:38.497524 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:26:38.497452 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73\": container with ID starting with ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73 not found: ID does not exist" containerID="ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73" Jul 30 17:26:38.497524 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.497493 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73"} err="failed to get container status \"ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73\": rpc error: code = NotFound desc = could not find container \"ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73\": container with ID starting with ca52de2aee1c3dacd1e10cc16807082623b171fc6f102e08103871cbf2f76a73 not found: ID does not exist" Jul 30 17:26:38.497668 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.497526 2576 scope.go:117] "RemoveContainer" containerID="c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e" Jul 30 17:26:38.497804 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:26:38.497788 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e\": container with ID starting with c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e not found: ID does not exist" containerID="c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e" Jul 30 17:26:38.497849 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.497811 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e"} err="failed to get container status \"c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e\": rpc error: code = NotFound desc = could not find container \"c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e\": container with ID starting with c774c68e5dfa9097af9c73cdf7e38ab11e69ff279fe21844e7f206dadefe4f6e not found: ID does not exist" Jul 30 17:26:38.497849 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.497826 2576 scope.go:117] "RemoveContainer" containerID="12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a" Jul 30 17:26:38.498075 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:26:38.498053 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a\": container with ID starting with 12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a not found: ID does not exist" containerID="12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a" Jul 30 17:26:38.498152 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.498083 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a"} err="failed to get container status \"12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a\": rpc error: code = NotFound desc = could not find container \"12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a\": container with ID starting with 12aff2ef651daa9473711f0c845b258a6813493e0ee41a0437702ef54cb62c5a not found: ID does not exist" Jul 30 17:26:38.498717 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.498700 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472"] Jul 30 17:26:38.508708 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.508680 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-ts472"] Jul 30 17:26:38.937966 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:38.937935 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c" path="/var/lib/kubelet/pods/6e0c6d2d-85f3-4d03-b8aa-738f435c5a6c/volumes" Jul 30 17:26:39.478436 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:39.478403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerStarted","Data":"88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065"} Jul 30 17:26:40.498588 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:40.498552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:26:42.492958 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:42.492924 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerID="88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065" exitCode=0 Jul 30 17:26:42.493334 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:42.492983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerDied","Data":"88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065"} Jul 30 17:26:43.499337 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:43.499305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerStarted","Data":"4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151"} Jul 30 17:26:43.499774 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:43.499364 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerStarted","Data":"e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7"} Jul 30 17:26:43.499774 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:43.499683 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:43.499857 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:43.499821 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:43.501329 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:43.501304 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:26:43.529800 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:43.529747 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podStartSLOduration=2.956895085 podStartE2EDuration="6.529731345s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.57283626s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:26:43.528536985 +0000 UTC m=+1573.201734462" watchObservedRunningTime="2026-07-30 17:26:43.529731345 +0000 UTC m=+1573.202928822" Jul 30 17:26:44.505004 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:44.504962 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:26:49.509537 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:49.509510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:26:49.509990 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:49.509958 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:26:59.510691 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:26:59.510650 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:27:09.510770 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:27:09.510733 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:27:19.510925 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:27:19.510884 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:27:29.510255 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:27:29.510176 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:27:39.510118 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:27:39.510075 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:27:49.510933 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:27:49.510894 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:27:59.510523 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:27:59.510494 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:28:08.078737 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.078698 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd"] Jul 30 17:28:08.079126 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.078938 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" containerID="cri-o://e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7" gracePeriod=30 Jul 30 17:28:08.079126 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.078977 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kube-rbac-proxy" containerID="cri-o://4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151" gracePeriod=30 Jul 30 17:28:08.096465 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.096436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:28:08.219281 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.219242 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd"] Jul 30 17:28:08.222863 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.222842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.228115 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.228091 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Jul 30 17:28:08.238868 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.238844 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd"] Jul 30 17:28:08.277727 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.277696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.277885 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.277750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.277885 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.277811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2bs\" (UniqueName: \"kubernetes.io/projected/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-api-access-rv2bs\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.277885 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.277833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.277885 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.277855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.378439 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.378346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2bs\" (UniqueName: \"kubernetes.io/projected/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-api-access-rv2bs\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.378439 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.378420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.378636 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.378445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.378636 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.378490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.378636 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.378552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.378876 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.378854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.379245 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.379217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.379351 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.379222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.381028 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.381009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.391488 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.391469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2bs\" (UniqueName: \"kubernetes.io/projected/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-api-access-rv2bs\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-7tkkd\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.533075 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.533038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:08.670922 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.670797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd"] Jul 30 17:28:08.673493 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:28:08.673468 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f878d65_257e_4db8_9f02_2ebfac2bd2b7.slice/crio-0f3235a5cbf0a79731835a03e15f9c33850d172475a9a840da27020f8e18e2b5 WatchSource:0}: Error finding container 0f3235a5cbf0a79731835a03e15f9c33850d172475a9a840da27020f8e18e2b5: Status 404 returned error can't find the container with id 0f3235a5cbf0a79731835a03e15f9c33850d172475a9a840da27020f8e18e2b5 Jul 30 17:28:08.874521 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.874482 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerID="4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151" exitCode=2 Jul 30 17:28:08.874678 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.874559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerDied","Data":"4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151"} Jul 30 17:28:08.876119 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.876094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerStarted","Data":"ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b"} Jul 30 17:28:08.876228 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:08.876149 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerStarted","Data":"0f3235a5cbf0a79731835a03e15f9c33850d172475a9a840da27020f8e18e2b5"} Jul 30 17:28:09.505918 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:09.505875 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused" Jul 30 17:28:09.510209 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:09.510187 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Jul 30 17:28:12.818245 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.818219 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:28:12.894077 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.894005 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerID="ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b" exitCode=0 Jul 30 17:28:12.894202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.894083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerDied","Data":"ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b"} Jul 30 17:28:12.896126 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.896106 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerID="e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7" exitCode=0 Jul 30 17:28:12.896224 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.896139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerDied","Data":"e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7"} Jul 30 17:28:12.896224 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.896158 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" Jul 30 17:28:12.896224 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.896176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd" event={"ID":"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d","Type":"ContainerDied","Data":"8f0171ebb72a889a7543e5f098dfe1ea02ae3a2bf0fc28b1097d06469a6aad36"} Jul 30 17:28:12.897202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.896198 2576 scope.go:117] "RemoveContainer" containerID="4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151" Jul 30 17:28:12.907671 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.907649 2576 scope.go:117] "RemoveContainer" containerID="e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7" Jul 30 17:28:12.916893 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.916870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmhw6\" (UniqueName: \"kubernetes.io/projected/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-api-access-zmhw6\") pod \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " Jul 30 17:28:12.917037 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.916922 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-rbac-proxy-sar-config\") pod \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " Jul 30 17:28:12.917037 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.917018 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-cabundle-cert\") pod \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " Jul 30 17:28:12.917152 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.917075 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kserve-provision-location\") pod \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " Jul 30 17:28:12.917152 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.917107 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-proxy-tls\") pod \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\" (UID: \"a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d\") " Jul 30 17:28:12.917554 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.917341 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" (UID: "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:28:12.917554 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.917468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" (UID: "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:28:12.917554 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.917486 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:28:12.917902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.917856 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" (UID: "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 17:28:12.918982 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.918960 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-api-access-zmhw6" (OuterVolumeSpecName: "kube-api-access-zmhw6") pod "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" (UID: "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d"). InnerVolumeSpecName "kube-api-access-zmhw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:28:12.919328 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.919305 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" (UID: "a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:28:12.919511 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.919496 2576 scope.go:117] "RemoveContainer" containerID="88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065" Jul 30 17:28:12.932078 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.932056 2576 scope.go:117] "RemoveContainer" containerID="4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151" Jul 30 17:28:12.932342 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:28:12.932321 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151\": container with ID starting with 4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151 not found: ID does not exist" containerID="4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151" Jul 30 17:28:12.932424 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.932350 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151"} err="failed to get container status \"4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151\": rpc error: code = NotFound desc = could not find container \"4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151\": container with ID starting with 4d64391735acfe10944e40e4e7527457a8dc704899890e67338ffad61cbd7151 not found: ID does not exist" Jul 30 17:28:12.932424 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.932407 2576 scope.go:117] "RemoveContainer" containerID="e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7" Jul 30 17:28:12.932682 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:28:12.932664 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7\": container with ID starting with e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7 not found: ID does not exist" containerID="e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7" Jul 30 17:28:12.932744 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.932687 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7"} err="failed to get container status \"e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7\": rpc error: code = NotFound desc = could not find container \"e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7\": container with ID starting with e34f8c093f1c9012703dc144177610a703ae937dbcb83fd094dfa0568658e8d7 not found: ID does not exist" Jul 30 17:28:12.932744 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.932702 2576 scope.go:117] "RemoveContainer" containerID="88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065" Jul 30 17:28:12.932975 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:28:12.932959 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065\": container with ID starting with 88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065 not found: ID does not exist" containerID="88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065" Jul 30 17:28:12.933035 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:12.932978 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065"} err="failed to get container status \"88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065\": rpc error: code = NotFound desc = could not find container \"88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065\": container with ID starting with 88bfd06e3ec9ae3dfef7afa9619585c03367f17b471d21bf692a42ed5eb1d065 not found: ID does not exist" Jul 30 17:28:13.018730 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.018710 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhw6\" (UniqueName: \"kubernetes.io/projected/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kube-api-access-zmhw6\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:28:13.018809 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.018731 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-cabundle-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:28:13.018809 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.018742 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-kserve-provision-location\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:28:13.018809 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.018753 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d-proxy-tls\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:28:13.219871 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.219838 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd"] Jul 30 17:28:13.226716 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.226691 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-6fnpd"] Jul 30 17:28:13.903953 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.903921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerStarted","Data":"238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09"} Jul 30 17:28:13.904316 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.903989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerStarted","Data":"aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d"} Jul 30 17:28:13.904316 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.904194 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:13.942159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:13.942065 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podStartSLOduration=2.000863586 podStartE2EDuration="5.942050766s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.94118718s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:28:13.938506465 +0000 UTC m=+1663.611703941" watchObservedRunningTime="2026-07-30 17:28:13.942050766 +0000 UTC m=+1663.615248242" Jul 30 17:28:14.909096 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:14.909068 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:14.910208 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:14.910181 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:28:14.938381 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:14.938341 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" path="/var/lib/kubelet/pods/a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d/volumes" Jul 30 17:28:15.913643 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:15.913600 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:28:20.917946 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:20.917914 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:28:20.918590 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:20.918558 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:28:30.918805 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:30.918767 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:28:40.918785 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:40.918747 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:28:50.918536 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:28:50.918500 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:29:00.919271 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:00.919190 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:29:10.918674 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:10.918632 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:29:20.918933 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:20.918887 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:29:30.919982 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:30.919952 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:29:38.260311 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:38.260270 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd"] Jul 30 17:29:38.260738 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:38.260557 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" containerID="cri-o://aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d" gracePeriod=30 Jul 30 17:29:38.260738 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:38.260586 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kube-rbac-proxy" containerID="cri-o://238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09" gracePeriod=30 Jul 30 17:29:38.311002 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:38.310969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:29:39.277719 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:39.277684 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerID="238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09" exitCode=2 Jul 30 17:29:39.278107 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:39.277759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerDied","Data":"238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09"} Jul 30 17:29:40.914359 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:40.914309 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused" Jul 30 17:29:40.918871 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:40.918848 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused" Jul 30 17:29:42.799814 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.799789 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:29:42.908201 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908113 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-rbac-proxy-sar-config\") pod \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " Jul 30 17:29:42.908201 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv2bs\" (UniqueName: \"kubernetes.io/projected/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-api-access-rv2bs\") pod \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " Jul 30 17:29:42.908201 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908169 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-proxy-tls\") pod \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " Jul 30 17:29:42.908524 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908248 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kserve-provision-location\") pod \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " Jul 30 17:29:42.908524 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908267 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-cabundle-cert\") pod \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\" (UID: \"8f878d65-257e-4db8-9f02-2ebfac2bd2b7\") " Jul 30 17:29:42.908619 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908594 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f878d65-257e-4db8-9f02-2ebfac2bd2b7" (UID: "8f878d65-257e-4db8-9f02-2ebfac2bd2b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 17:29:42.908656 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908639 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f878d65-257e-4db8-9f02-2ebfac2bd2b7" (UID: "8f878d65-257e-4db8-9f02-2ebfac2bd2b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:29:42.908788 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.908758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f878d65-257e-4db8-9f02-2ebfac2bd2b7" (UID: "8f878d65-257e-4db8-9f02-2ebfac2bd2b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:29:42.910274 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.910246 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f878d65-257e-4db8-9f02-2ebfac2bd2b7" (UID: "8f878d65-257e-4db8-9f02-2ebfac2bd2b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:29:42.910411 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:42.910389 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-api-access-rv2bs" (OuterVolumeSpecName: "kube-api-access-rv2bs") pod "8f878d65-257e-4db8-9f02-2ebfac2bd2b7" (UID: "8f878d65-257e-4db8-9f02-2ebfac2bd2b7"). InnerVolumeSpecName "kube-api-access-rv2bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:29:43.009631 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.009600 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kserve-provision-location\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:29:43.009631 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.009627 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-cabundle-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:29:43.009631 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.009636 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:29:43.009828 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.009646 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv2bs\" (UniqueName: \"kubernetes.io/projected/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-kube-api-access-rv2bs\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:29:43.009828 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.009655 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f878d65-257e-4db8-9f02-2ebfac2bd2b7-proxy-tls\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:29:43.295921 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.295889 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerID="aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d" exitCode=0 Jul 30 17:29:43.296085 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.295945 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" Jul 30 17:29:43.296085 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.295971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerDied","Data":"aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d"} Jul 30 17:29:43.296085 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.296031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd" event={"ID":"8f878d65-257e-4db8-9f02-2ebfac2bd2b7","Type":"ContainerDied","Data":"0f3235a5cbf0a79731835a03e15f9c33850d172475a9a840da27020f8e18e2b5"} Jul 30 17:29:43.296085 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.296059 2576 scope.go:117] "RemoveContainer" containerID="238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09" Jul 30 17:29:43.304354 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.304330 2576 scope.go:117] "RemoveContainer" containerID="aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d" Jul 30 17:29:43.311788 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.311769 2576 scope.go:117] "RemoveContainer" containerID="ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b" Jul 30 17:29:43.319159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.319145 2576 scope.go:117] "RemoveContainer" containerID="238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09" Jul 30 17:29:43.319423 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:29:43.319405 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09\": container with ID starting with 238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09 not found: ID does not exist" containerID="238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09" Jul 30 17:29:43.319489 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.319429 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09"} err="failed to get container status \"238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09\": rpc error: code = NotFound desc = could not find container \"238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09\": container with ID starting with 238761d1a5c809b9643bd5fc55fee64d533dc07d806570f410c98a892fba3e09 not found: ID does not exist" Jul 30 17:29:43.319489 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.319444 2576 scope.go:117] "RemoveContainer" containerID="aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d" Jul 30 17:29:43.319675 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:29:43.319658 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d\": container with ID starting with aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d not found: ID does not exist" containerID="aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d" Jul 30 17:29:43.319719 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.319681 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d"} err="failed to get container status \"aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d\": rpc error: code = NotFound desc = could not find container \"aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d\": container with ID starting with aa43215cc5a8a0ed4fd28824e25224cb3d4375627acb71a1eed6ac0099f67e9d not found: ID does not exist" Jul 30 17:29:43.319719 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.319697 2576 scope.go:117] "RemoveContainer" containerID="ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b" Jul 30 17:29:43.319945 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:29:43.319927 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b\": container with ID starting with ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b not found: ID does not exist" containerID="ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b" Jul 30 17:29:43.320003 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.319956 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b"} err="failed to get container status \"ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b\": rpc error: code = NotFound desc = could not find container \"ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b\": container with ID starting with ebcf5ee6eb10b336582bff8330ccd64d06f95573eeb194cef35d9af2e27f094b not found: ID does not exist" Jul 30 17:29:43.347648 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.347624 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd"] Jul 30 17:29:43.353603 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.353582 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-7tkkd"] Jul 30 17:29:43.494805 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.494749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:29:43.577607 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.577500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:29:43.579307 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579282 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs"] Jul 30 17:29:43.579679 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579662 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="storage-initializer" Jul 30 17:29:43.579757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579681 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="storage-initializer" Jul 30 17:29:43.579757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579708 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" Jul 30 17:29:43.579757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579719 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" Jul 30 17:29:43.579757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579729 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" Jul 30 17:29:43.579757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579737 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" Jul 30 17:29:43.579757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579751 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kube-rbac-proxy" Jul 30 17:29:43.579757 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579759 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kube-rbac-proxy" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579782 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="storage-initializer" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579791 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="storage-initializer" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579802 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kube-rbac-proxy" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579810 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kube-rbac-proxy" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579882 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kube-rbac-proxy" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579898 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" containerName="kserve-container" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579908 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kube-rbac-proxy" Jul 30 17:29:43.580097 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.579918 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9a8b32a-4eea-44f3-95dc-dfbdc6f61e0d" containerName="kserve-container" Jul 30 17:29:43.584358 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.584339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.587459 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.587440 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 30 17:29:43.588077 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.588058 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Jul 30 17:29:43.588186 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.588102 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qz9j4\"" Jul 30 17:29:43.588186 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.588150 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 30 17:29:43.588300 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.588244 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Jul 30 17:29:43.588366 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.588351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 30 17:29:43.588493 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.588474 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 30 17:29:43.594445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.594423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs"] Jul 30 17:29:43.715060 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.715016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.715253 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.715076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-cabundle-cert\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.715253 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.715108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4w7p\" (UniqueName: \"kubernetes.io/projected/99428d45-312f-4269-a0fd-9dad31e482de-kube-api-access-s4w7p\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.715253 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.715199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.715422 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.715259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99428d45-312f-4269-a0fd-9dad31e482de-kserve-provision-location\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.816592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.816556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-cabundle-cert\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.816592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.816598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4w7p\" (UniqueName: \"kubernetes.io/projected/99428d45-312f-4269-a0fd-9dad31e482de-kube-api-access-s4w7p\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.817090 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.816700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.817090 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.816726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99428d45-312f-4269-a0fd-9dad31e482de-kserve-provision-location\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.817090 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.816769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.817090 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:29:43.816873 2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found Jul 30 17:29:43.817090 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:29:43.816939 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls podName:99428d45-312f-4269-a0fd-9dad31e482de nodeName:}" failed. No retries permitted until 2026-07-30 17:29:44.316918864 +0000 UTC m=+1753.990116320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls") pod "isvc-canary-stop-predictor-77665d5899-vhszs" (UID: "99428d45-312f-4269-a0fd-9dad31e482de") : secret "isvc-canary-stop-predictor-serving-cert" not found Jul 30 17:29:43.817295 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.817271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99428d45-312f-4269-a0fd-9dad31e482de-kserve-provision-location\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.817360 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.817339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-cabundle-cert\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.817421 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.817398 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.830032 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.829973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4w7p\" (UniqueName: \"kubernetes.io/projected/99428d45-312f-4269-a0fd-9dad31e482de-kube-api-access-s4w7p\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:43.902198 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:43.902147 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:29:44.314545 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:44.314512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:29:44.320634 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:44.320609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:44.322995 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:44.322966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-vhszs\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:44.496621 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:44.496582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:44.626492 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:44.626468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs"] Jul 30 17:29:44.628681 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:29:44.628656 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99428d45_312f_4269_a0fd_9dad31e482de.slice/crio-1e91d7011e3b590ca4d9c184a05c6db20df007dc9715ca39795a3fed107ebac5 WatchSource:0}: Error finding container 1e91d7011e3b590ca4d9c184a05c6db20df007dc9715ca39795a3fed107ebac5: Status 404 returned error can't find the container with id 1e91d7011e3b590ca4d9c184a05c6db20df007dc9715ca39795a3fed107ebac5 Jul 30 17:29:44.939942 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:44.939853 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f878d65-257e-4db8-9f02-2ebfac2bd2b7" path="/var/lib/kubelet/pods/8f878d65-257e-4db8-9f02-2ebfac2bd2b7/volumes" Jul 30 17:29:45.307268 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:45.307239 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerStarted","Data":"8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a"} Jul 30 17:29:45.307459 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:45.307308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerStarted","Data":"1e91d7011e3b590ca4d9c184a05c6db20df007dc9715ca39795a3fed107ebac5"} Jul 30 17:29:49.324156 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:49.324123 2576 generic.go:356] "Generic (PLEG): container finished" podID="99428d45-312f-4269-a0fd-9dad31e482de" containerID="8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a" exitCode=0 Jul 30 17:29:49.324625 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:49.324197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerDied","Data":"8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a"} Jul 30 17:29:50.330643 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:50.330608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerStarted","Data":"00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a"} Jul 30 17:29:50.331001 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:50.330700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerStarted","Data":"4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2"} Jul 30 17:29:50.331001 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:50.330963 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:50.356278 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:50.356226 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podStartSLOduration=3.476689032 podStartE2EDuration="7.356210584s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.879521552s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:29:50.354721062 +0000 UTC m=+1760.027918562" watchObservedRunningTime="2026-07-30 17:29:50.356210584 +0000 UTC m=+1760.029408061" Jul 30 17:29:51.335529 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:51.335491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:51.336514 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:51.336494 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:29:52.339750 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:52.339707 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:29:57.344872 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:57.344841 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:29:57.345443 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:29:57.345409 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:30:07.346220 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:30:07.346176 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:30:17.345628 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:30:17.345576 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:30:27.345427 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:30:27.345319 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:30:37.346274 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:30:37.346236 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:30:47.345894 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:30:47.345850 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:30:57.345412 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:30:57.345355 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Jul 30 17:31:07.346041 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:07.346000 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:31:18.609960 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:18.609925 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs"] Jul 30 17:31:18.610496 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:18.610201 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" containerID="cri-o://4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2" gracePeriod=30 Jul 30 17:31:18.610496 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:18.610261 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kube-rbac-proxy" containerID="cri-o://00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a" gracePeriod=30 Jul 30 17:31:18.663758 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:18.663728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:31:19.707252 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:19.707221 2576 generic.go:356] "Generic (PLEG): container finished" podID="99428d45-312f-4269-a0fd-9dad31e482de" containerID="00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a" exitCode=2 Jul 30 17:31:19.707694 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:19.707304 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerDied","Data":"00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a"} Jul 30 17:31:19.886288 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:19.886240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:31:22.340701 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:22.340660 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused" Jul 30 17:31:23.357447 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.357421 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:31:23.430671 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.430590 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99428d45-312f-4269-a0fd-9dad31e482de-kserve-provision-location\") pod \"99428d45-312f-4269-a0fd-9dad31e482de\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " Jul 30 17:31:23.430671 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.430664 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-cabundle-cert\") pod \"99428d45-312f-4269-a0fd-9dad31e482de\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " Jul 30 17:31:23.430902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.430699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-kube-rbac-proxy-sar-config\") pod \"99428d45-312f-4269-a0fd-9dad31e482de\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " Jul 30 17:31:23.430902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.430734 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls\") pod \"99428d45-312f-4269-a0fd-9dad31e482de\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " Jul 30 17:31:23.430902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.430849 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4w7p\" (UniqueName: \"kubernetes.io/projected/99428d45-312f-4269-a0fd-9dad31e482de-kube-api-access-s4w7p\") pod \"99428d45-312f-4269-a0fd-9dad31e482de\" (UID: \"99428d45-312f-4269-a0fd-9dad31e482de\") " Jul 30 17:31:23.430902 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.430890 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99428d45-312f-4269-a0fd-9dad31e482de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99428d45-312f-4269-a0fd-9dad31e482de" (UID: "99428d45-312f-4269-a0fd-9dad31e482de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 17:31:23.431115 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.431066 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99428d45-312f-4269-a0fd-9dad31e482de" (UID: "99428d45-312f-4269-a0fd-9dad31e482de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:31:23.431115 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.431080 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99428d45-312f-4269-a0fd-9dad31e482de" (UID: "99428d45-312f-4269-a0fd-9dad31e482de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:31:23.431183 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.431158 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-cabundle-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:31:23.431183 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.431174 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99428d45-312f-4269-a0fd-9dad31e482de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:31:23.431245 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.431183 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99428d45-312f-4269-a0fd-9dad31e482de-kserve-provision-location\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:31:23.432856 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.432835 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99428d45-312f-4269-a0fd-9dad31e482de-kube-api-access-s4w7p" (OuterVolumeSpecName: "kube-api-access-s4w7p") pod "99428d45-312f-4269-a0fd-9dad31e482de" (UID: "99428d45-312f-4269-a0fd-9dad31e482de"). InnerVolumeSpecName "kube-api-access-s4w7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:31:23.432856 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.432842 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99428d45-312f-4269-a0fd-9dad31e482de" (UID: "99428d45-312f-4269-a0fd-9dad31e482de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:31:23.532457 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.532410 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99428d45-312f-4269-a0fd-9dad31e482de-proxy-tls\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:31:23.532457 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.532452 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4w7p\" (UniqueName: \"kubernetes.io/projected/99428d45-312f-4269-a0fd-9dad31e482de-kube-api-access-s4w7p\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:31:23.726000 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.725968 2576 generic.go:356] "Generic (PLEG): container finished" podID="99428d45-312f-4269-a0fd-9dad31e482de" containerID="4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2" exitCode=0 Jul 30 17:31:23.726169 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.726041 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" Jul 30 17:31:23.726169 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.726054 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerDied","Data":"4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2"} Jul 30 17:31:23.726169 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.726114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs" event={"ID":"99428d45-312f-4269-a0fd-9dad31e482de","Type":"ContainerDied","Data":"1e91d7011e3b590ca4d9c184a05c6db20df007dc9715ca39795a3fed107ebac5"} Jul 30 17:31:23.726169 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.726141 2576 scope.go:117] "RemoveContainer" containerID="00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a" Jul 30 17:31:23.735985 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.735962 2576 scope.go:117] "RemoveContainer" containerID="4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2" Jul 30 17:31:23.748398 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.743659 2576 scope.go:117] "RemoveContainer" containerID="8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a" Jul 30 17:31:23.755719 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.755698 2576 scope.go:117] "RemoveContainer" containerID="00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a" Jul 30 17:31:23.755955 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:31:23.755934 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a\": container with ID starting with 00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a not found: ID does not exist" containerID="00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a" Jul 30 17:31:23.756028 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.755962 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a"} err="failed to get container status \"00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a\": rpc error: code = NotFound desc = could not find container \"00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a\": container with ID starting with 00675e132b747094fe7ab58ecfb3bc816bb897b2bcca5a352447ce54636cfb3a not found: ID does not exist" Jul 30 17:31:23.756028 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.755979 2576 scope.go:117] "RemoveContainer" containerID="4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2" Jul 30 17:31:23.756194 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:31:23.756178 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2\": container with ID starting with 4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2 not found: ID does not exist" containerID="4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2" Jul 30 17:31:23.756236 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.756197 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2"} err="failed to get container status \"4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2\": rpc error: code = NotFound desc = could not find container \"4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2\": container with ID starting with 4706ba31d9b836211a71e5b6e31d97928f52329c812c44a6038d927d59557ed2 not found: ID does not exist" Jul 30 17:31:23.756236 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.756210 2576 scope.go:117] "RemoveContainer" containerID="8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a" Jul 30 17:31:23.756433 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:31:23.756413 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a\": container with ID starting with 8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a not found: ID does not exist" containerID="8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a" Jul 30 17:31:23.756485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.756440 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a"} err="failed to get container status \"8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a\": rpc error: code = NotFound desc = could not find container \"8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a\": container with ID starting with 8ce70a70783a8c5032326cdf8fc9f0f96ae43b4a80e8c38d7372016a077efa8a not found: ID does not exist" Jul 30 17:31:23.771079 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.771045 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs"] Jul 30 17:31:23.779253 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:23.779229 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-vhszs"] Jul 30 17:31:24.938345 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:24.938309 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99428d45-312f-4269-a0fd-9dad31e482de" path="/var/lib/kubelet/pods/99428d45-312f-4269-a0fd-9dad31e482de/volumes" Jul 30 17:31:28.911509 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:28.911481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:31:29.220385 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:31:29.220335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:32:00.947736 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:32:00.947642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:14.175567 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:14.175523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:14.315138 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:14.315106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:32.324791 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:32.324702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:32.418903 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:32.418875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:37.448121 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:37.448086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:37.737781 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:37.737754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:42.541433 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:42.541400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:47.647718 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.647680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:47.693670 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.693636 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2"] Jul 30 17:33:47.694112 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694097 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="storage-initializer" Jul 30 17:33:47.694159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694115 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="storage-initializer" Jul 30 17:33:47.694159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694135 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" Jul 30 17:33:47.694159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694144 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" Jul 30 17:33:47.694159 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694156 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kube-rbac-proxy" Jul 30 17:33:47.694296 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694165 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kube-rbac-proxy" Jul 30 17:33:47.694296 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694235 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kube-rbac-proxy" Jul 30 17:33:47.694296 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.694247 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="99428d45-312f-4269-a0fd-9dad31e482de" containerName="kserve-container" Jul 30 17:33:47.697843 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.697826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.700334 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.700306 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b3e6c-predictor-serving-cert\"" Jul 30 17:33:47.700485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.700397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 30 17:33:47.700485 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.700397 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b3e6c-kube-rbac-proxy-sar-config\"" Jul 30 17:33:47.700604 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.700586 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 30 17:33:47.700790 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.700764 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 30 17:33:47.701424 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.701410 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qz9j4\"" Jul 30 17:33:47.701470 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.701440 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 30 17:33:47.709917 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.709894 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2"] Jul 30 17:33:47.773322 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.773285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-proxy-tls\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.773518 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.773335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj44k\" (UniqueName: \"kubernetes.io/projected/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-api-access-fj44k\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.773518 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.773426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.773518 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.773486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-cabundle-cert\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.773630 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.773529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kserve-provision-location\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.790025 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.789998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:47.874970 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.874938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-cabundle-cert\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.875144 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.874977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kserve-provision-location\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.875144 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.875003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-proxy-tls\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.875144 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.875038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj44k\" (UniqueName: \"kubernetes.io/projected/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-api-access-fj44k\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.875144 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.875063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.875432 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.875414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kserve-provision-location\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.875850 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.875829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.875906 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.875829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-cabundle-cert\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.877504 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.877484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-proxy-tls\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:47.883771 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:47.883743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj44k\" (UniqueName: \"kubernetes.io/projected/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-api-access-fj44k\") pod \"raw-sklearn-b3e6c-predictor-5748869db6-phsr2\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:48.010560 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:48.010525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:48.144044 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:48.144020 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2"] Jul 30 17:33:48.146004 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:33:48.145975 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d6d2cb_0c91_4854_8eac_0db1cdb0fce6.slice/crio-bed92b156b610b781a34044b940b645e1e166eb4899a660b2c8d08e85cd1304d WatchSource:0}: Error finding container bed92b156b610b781a34044b940b645e1e166eb4899a660b2c8d08e85cd1304d: Status 404 returned error can't find the container with id bed92b156b610b781a34044b940b645e1e166eb4899a660b2c8d08e85cd1304d Jul 30 17:33:48.310247 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:48.310162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerStarted","Data":"be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508"} Jul 30 17:33:48.310247 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:48.310228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerStarted","Data":"bed92b156b610b781a34044b940b645e1e166eb4899a660b2c8d08e85cd1304d"} Jul 30 17:33:52.327716 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:52.327686 2576 generic.go:356] "Generic (PLEG): container finished" podID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerID="be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508" exitCode=0 Jul 30 17:33:52.328202 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:52.327749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerDied","Data":"be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508"} Jul 30 17:33:53.333938 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:53.333900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerStarted","Data":"098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47"} Jul 30 17:33:53.334399 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:53.333964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerStarted","Data":"138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05"} Jul 30 17:33:53.334399 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:53.334240 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:53.334491 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:53.334398 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:53.335645 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:53.335622 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:33:53.358976 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:53.358931 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podStartSLOduration=2.7001989010000003 podStartE2EDuration="6.358916392s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.658717491s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:33:53.357639265 +0000 UTC m=+2003.030836741" watchObservedRunningTime="2026-07-30 17:33:53.358916392 +0000 UTC m=+2003.032113871" Jul 30 17:33:54.337690 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:54.337652 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:33:55.430908 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:55.430876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:33:59.342959 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:59.342929 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:33:59.343605 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:33:59.343557 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:34:09.344144 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:34:09.344099 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:34:19.343586 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:34:19.343541 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:34:29.343573 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:34:29.343524 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:34:39.343509 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:34:39.343473 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:34:49.343578 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:34:49.343533 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:34:59.343759 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:34:59.343673 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:35:09.344140 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:09.344109 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:35:17.840481 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:17.840443 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2"] Jul 30 17:35:17.840929 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:17.840696 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" containerID="cri-o://138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05" gracePeriod=30 Jul 30 17:35:17.840929 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:17.840786 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kube-rbac-proxy" containerID="cri-o://098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47" gracePeriod=30 Jul 30 17:35:17.854686 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:17.854661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:35:18.691201 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:18.691163 2576 generic.go:356] "Generic (PLEG): container finished" podID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerID="098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47" exitCode=2 Jul 30 17:35:18.691592 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:18.691234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerDied","Data":"098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47"} Jul 30 17:35:19.338420 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:19.338349 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused" Jul 30 17:35:19.343953 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:19.343923 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused" Jul 30 17:35:22.684051 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.684027 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:35:22.715470 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.715359 2576 generic.go:356] "Generic (PLEG): container finished" podID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerID="138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05" exitCode=0 Jul 30 17:35:22.715470 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.715407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerDied","Data":"138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05"} Jul 30 17:35:22.715470 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.715443 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" Jul 30 17:35:22.715730 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.715489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2" event={"ID":"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6","Type":"ContainerDied","Data":"bed92b156b610b781a34044b940b645e1e166eb4899a660b2c8d08e85cd1304d"} Jul 30 17:35:22.715730 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.715517 2576 scope.go:117] "RemoveContainer" containerID="098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47" Jul 30 17:35:22.725572 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.725553 2576 scope.go:117] "RemoveContainer" containerID="138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05" Jul 30 17:35:22.732846 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.732830 2576 scope.go:117] "RemoveContainer" containerID="be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508" Jul 30 17:35:22.739621 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.739605 2576 scope.go:117] "RemoveContainer" containerID="098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47" Jul 30 17:35:22.739894 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:35:22.739868 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47\": container with ID starting with 098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47 not found: ID does not exist" containerID="098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47" Jul 30 17:35:22.739944 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.739895 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47"} err="failed to get container status \"098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47\": rpc error: code = NotFound desc = could not find container \"098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47\": container with ID starting with 098b5b500bf41c4894e72e8393794f59eb4bfa7aa6b1190d79afb26c18b17e47 not found: ID does not exist" Jul 30 17:35:22.739944 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.739916 2576 scope.go:117] "RemoveContainer" containerID="138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05" Jul 30 17:35:22.740131 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:35:22.740116 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05\": container with ID starting with 138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05 not found: ID does not exist" containerID="138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05" Jul 30 17:35:22.740179 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.740138 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05"} err="failed to get container status \"138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05\": rpc error: code = NotFound desc = could not find container \"138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05\": container with ID starting with 138df6d3bbb6e304246a7ed9d68928491d4a84f3cdbbe8a4477c89390c3fad05 not found: ID does not exist" Jul 30 17:35:22.740179 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.740152 2576 scope.go:117] "RemoveContainer" containerID="be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508" Jul 30 17:35:22.740385 ip-10-0-130-254 kubenswrapper[2576]: E0730 17:35:22.740360 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508\": container with ID starting with be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508 not found: ID does not exist" containerID="be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508" Jul 30 17:35:22.740427 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.740395 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508"} err="failed to get container status \"be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508\": rpc error: code = NotFound desc = could not find container \"be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508\": container with ID starting with be35ac2c55835c3a7fc312df30296bac88acd04dd32e4c2dea14b28a358d2508 not found: ID does not exist" Jul 30 17:35:22.798855 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.798825 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-cabundle-cert\") pod \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " Jul 30 17:35:22.799006 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.798883 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj44k\" (UniqueName: \"kubernetes.io/projected/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-api-access-fj44k\") pod \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " Jul 30 17:35:22.799006 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.798953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-rbac-proxy-sar-config\") pod \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " Jul 30 17:35:22.799087 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.799005 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-proxy-tls\") pod \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " Jul 30 17:35:22.799087 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.799031 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kserve-provision-location\") pod \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\" (UID: \"33d6d2cb-0c91-4854-8eac-0db1cdb0fce6\") " Jul 30 17:35:22.799284 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.799256 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" (UID: "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:35:22.799426 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.799396 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" (UID: "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 17:35:22.799504 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.799478 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" (UID: "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 17:35:22.801104 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.801086 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" (UID: "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 17:35:22.801104 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.801098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-api-access-fj44k" (OuterVolumeSpecName: "kube-api-access-fj44k") pod "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" (UID: "33d6d2cb-0c91-4854-8eac-0db1cdb0fce6"). InnerVolumeSpecName "kube-api-access-fj44k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 17:35:22.900545 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.900509 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:35:22.900545 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.900541 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-proxy-tls\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:35:22.900545 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.900552 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kserve-provision-location\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:35:22.900822 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.900561 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-cabundle-cert\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:35:22.900822 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.900571 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj44k\" (UniqueName: \"kubernetes.io/projected/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6-kube-api-access-fj44k\") on node \"ip-10-0-130-254.ec2.internal\" DevicePath \"\"" Jul 30 17:35:22.957885 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:22.957852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:35:23.040640 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:23.040561 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2"] Jul 30 17:35:23.047022 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:23.046997 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b3e6c-predictor-5748869db6-phsr2"] Jul 30 17:35:23.084209 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:23.084183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:35:24.938474 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:24.938447 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" path="/var/lib/kubelet/pods/33d6d2cb-0c91-4854-8eac-0db1cdb0fce6/volumes" Jul 30 17:35:27.560080 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:35:27.560050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:36:12.911813 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:36:12.911779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:36:53.117881 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:36:53.117851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:36:59.020859 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:36:59.020826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:36:59.101432 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:36:59.101408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:37:28.518747 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.518713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:37:28.648916 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.648882 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5wwbm/must-gather-6df4f"] Jul 30 17:37:28.649210 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649198 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="storage-initializer" Jul 30 17:37:28.649261 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649212 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="storage-initializer" Jul 30 17:37:28.649261 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649223 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kube-rbac-proxy" Jul 30 17:37:28.649261 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649229 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kube-rbac-proxy" Jul 30 17:37:28.649261 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649240 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" Jul 30 17:37:28.649261 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649245 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" Jul 30 17:37:28.649445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649305 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kube-rbac-proxy" Jul 30 17:37:28.649445 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.649314 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d6d2cb-0c91-4854-8eac-0db1cdb0fce6" containerName="kserve-container" Jul 30 17:37:28.652158 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.652138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:28.654915 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.654890 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5wwbm\"/\"default-dockercfg-tg7fp\"" Jul 30 17:37:28.655052 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.654973 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wwbm\"/\"kube-root-ca.crt\"" Jul 30 17:37:28.655052 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.654988 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wwbm\"/\"openshift-service-ca.crt\"" Jul 30 17:37:28.662537 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.662516 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wwbm/must-gather-6df4f"] Jul 30 17:37:28.666700 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.666679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:37:28.751564 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.751538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zql8w\" (UniqueName: \"kubernetes.io/projected/d6167c7b-bb3e-46f7-b06d-6e08189e6f06-kube-api-access-zql8w\") pod \"must-gather-6df4f\" (UID: \"d6167c7b-bb3e-46f7-b06d-6e08189e6f06\") " pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:28.751704 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.751595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6167c7b-bb3e-46f7-b06d-6e08189e6f06-must-gather-output\") pod \"must-gather-6df4f\" (UID: \"d6167c7b-bb3e-46f7-b06d-6e08189e6f06\") " pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:28.766495 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.766473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:37:28.852365 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.852287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zql8w\" (UniqueName: \"kubernetes.io/projected/d6167c7b-bb3e-46f7-b06d-6e08189e6f06-kube-api-access-zql8w\") pod \"must-gather-6df4f\" (UID: \"d6167c7b-bb3e-46f7-b06d-6e08189e6f06\") " pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:28.852517 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.852361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6167c7b-bb3e-46f7-b06d-6e08189e6f06-must-gather-output\") pod \"must-gather-6df4f\" (UID: \"d6167c7b-bb3e-46f7-b06d-6e08189e6f06\") " pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:28.852756 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.852738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6167c7b-bb3e-46f7-b06d-6e08189e6f06-must-gather-output\") pod \"must-gather-6df4f\" (UID: \"d6167c7b-bb3e-46f7-b06d-6e08189e6f06\") " pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:28.862406 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.862387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zql8w\" (UniqueName: \"kubernetes.io/projected/d6167c7b-bb3e-46f7-b06d-6e08189e6f06-kube-api-access-zql8w\") pod \"must-gather-6df4f\" (UID: \"d6167c7b-bb3e-46f7-b06d-6e08189e6f06\") " pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:28.968647 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:28.968621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wwbm/must-gather-6df4f" Jul 30 17:37:29.094362 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:29.094338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wwbm/must-gather-6df4f"] Jul 30 17:37:29.097230 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:37:29.097203 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6167c7b_bb3e_46f7_b06d_6e08189e6f06.slice/crio-fe7257640347ff608aa62dee58b4178301caff43ba02966de9204f3353b45c7c WatchSource:0}: Error finding container fe7257640347ff608aa62dee58b4178301caff43ba02966de9204f3353b45c7c: Status 404 returned error can't find the container with id fe7257640347ff608aa62dee58b4178301caff43ba02966de9204f3353b45c7c Jul 30 17:37:29.099124 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:29.099106 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 17:37:29.223326 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:29.223293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wwbm/must-gather-6df4f" event={"ID":"d6167c7b-bb3e-46f7-b06d-6e08189e6f06","Type":"ContainerStarted","Data":"fe7257640347ff608aa62dee58b4178301caff43ba02966de9204f3353b45c7c"} Jul 30 17:37:30.232923 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:30.232844 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wwbm/must-gather-6df4f" event={"ID":"d6167c7b-bb3e-46f7-b06d-6e08189e6f06","Type":"ContainerStarted","Data":"a4ac890d63b24ec665a047e3b8310c16d0ab84edc6c6b2940050d046d05de3d1"} Jul 30 17:37:30.232923 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:30.232894 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wwbm/must-gather-6df4f" event={"ID":"d6167c7b-bb3e-46f7-b06d-6e08189e6f06","Type":"ContainerStarted","Data":"d07d92aa02f7707f2c31ae349267bef2ccca7f1ff8ffc35203caac17ceec1d08"} Jul 30 17:37:30.251652 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:30.251595 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5wwbm/must-gather-6df4f" podStartSLOduration=1.460659634 podStartE2EDuration="2.251574428s" totalImagesPullingTime="790.914794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:37:30.24979509 +0000 UTC m=+2219.922992580" watchObservedRunningTime="2026-07-30 17:37:30.251574428 +0000 UTC m=+2219.924771906" Jul 30 17:37:31.249389 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:31.249323 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9wmwb_479c1d63-717e-42f7-b325-e3585799cd58/global-pull-secret-syncer/0.log" Jul 30 17:37:31.411613 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:31.411584 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" Jul 30 17:37:31.546988 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:31.546917 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-254.ec2.internal_55e30e52bbfc0cb8ac949e33c9db102a/haproxy/0.log" Jul 30 17:37:34.755011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:34.754973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:37:34.843770 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:34.843741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hzs9_8eb802f4-7b82-453c-8997-d9ff1248192c/konnectivity-agent/0.log" limit=1024 Jul 30 17:37:35.512008 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:35.511975 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fc407a1-e51b-4652-a677-935c67c6a364/alertmanager/0.log" Jul 30 17:37:35.555346 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:35.555320 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fc407a1-e51b-4652-a677-935c67c6a364/config-reloader/0.log" Jul 30 17:37:35.608135 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:35.608109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fc407a1-e51b-4652-a677-935c67c6a364/kube-rbac-proxy-web/0.log" Jul 30 17:37:35.660143 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:35.660014 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fc407a1-e51b-4652-a677-935c67c6a364/kube-rbac-proxy/0.log" Jul 30 17:37:35.710420 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:35.710363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fc407a1-e51b-4652-a677-935c67c6a364/kube-rbac-proxy-metric/0.log" Jul 30 17:37:35.755278 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:35.755236 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fc407a1-e51b-4652-a677-935c67c6a364/prom-label-proxy/0.log" Jul 30 17:37:35.789032 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:35.788943 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9fc407a1-e51b-4652-a677-935c67c6a364/init-config-reloader/0.log" Jul 30 17:37:36.102631 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.102552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-jwfzd_c19a42a8-010d-4954-8b4e-af3e11414ce8/monitoring-plugin/0.log" Jul 30 17:37:36.157333 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.157300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsl2b_02922d64-a0c1-4fc4-91a8-b65655d2e369/node-exporter/0.log" Jul 30 17:37:36.199773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.199745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsl2b_02922d64-a0c1-4fc4-91a8-b65655d2e369/kube-rbac-proxy/0.log" Jul 30 17:37:36.234997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.234966 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jsl2b_02922d64-a0c1-4fc4-91a8-b65655d2e369/init-textfile/0.log" Jul 30 17:37:36.543842 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.543808 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-zc28c_620e2f01-559e-4eb2-9734-fc4721e2c53e/kube-rbac-proxy-main/0.log" Jul 30 17:37:36.588433 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.588399 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-zc28c_620e2f01-559e-4eb2-9734-fc4721e2c53e/kube-rbac-proxy-self/0.log" Jul 30 17:37:36.623986 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.623946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-zc28c_620e2f01-559e-4eb2-9734-fc4721e2c53e/openshift-state-metrics/0.log" Jul 30 17:37:36.685718 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.685684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d46b64b2-d813-4dbc-84cf-b80ce6f79769/prometheus/0.log" Jul 30 17:37:36.749011 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.748976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d46b64b2-d813-4dbc-84cf-b80ce6f79769/config-reloader/0.log" Jul 30 17:37:36.796943 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.796858 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d46b64b2-d813-4dbc-84cf-b80ce6f79769/thanos-sidecar/0.log" Jul 30 17:37:36.832802 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.832717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d46b64b2-d813-4dbc-84cf-b80ce6f79769/kube-rbac-proxy-web/0.log" Jul 30 17:37:36.874665 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.874593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d46b64b2-d813-4dbc-84cf-b80ce6f79769/kube-rbac-proxy/0.log" Jul 30 17:37:36.913849 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.913774 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d46b64b2-d813-4dbc-84cf-b80ce6f79769/kube-rbac-proxy-thanos/0.log" Jul 30 17:37:36.956186 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:36.956158 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d46b64b2-d813-4dbc-84cf-b80ce6f79769/init-config-reloader/0.log" Jul 30 17:37:37.172887 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:37.172858 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-555c98fb9b-6w9nj_ad4fd300-c9e2-4760-902c-40af264af1f3/telemeter-client/0.log" Jul 30 17:37:37.236257 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:37.236227 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-555c98fb9b-6w9nj_ad4fd300-c9e2-4760-902c-40af264af1f3/reload/0.log" Jul 30 17:37:37.323979 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:37.323734 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-555c98fb9b-6w9nj_ad4fd300-c9e2-4760-902c-40af264af1f3/kube-rbac-proxy/0.log" Jul 30 17:37:39.317284 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.317229 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q"] Jul 30 17:37:39.331152 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.331121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.333059 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.333025 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q"] Jul 30 17:37:39.457586 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.457551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wbp\" (UniqueName: \"kubernetes.io/projected/9f192353-fbca-4aea-a7e8-630edeb1c550-kube-api-access-c2wbp\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.457773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.457619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-lib-modules\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.457773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.457641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-proc\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.457773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.457676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-sys\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.457773 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.457690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-podres\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.558513 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-proc\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.558745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-sys\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.558745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-sys\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.558745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-proc\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.558745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-podres\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.558745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wbp\" (UniqueName: \"kubernetes.io/projected/9f192353-fbca-4aea-a7e8-630edeb1c550-kube-api-access-c2wbp\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.558745 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-lib-modules\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.559058 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-podres\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.559058 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.558838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f192353-fbca-4aea-a7e8-630edeb1c550-lib-modules\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.568921 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.568848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wbp\" (UniqueName: \"kubernetes.io/projected/9f192353-fbca-4aea-a7e8-630edeb1c550-kube-api-access-c2wbp\") pod \"perf-node-gather-daemonset-crd4q\" (UID: \"9f192353-fbca-4aea-a7e8-630edeb1c550\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.644961 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.644932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:39.792590 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.792548 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-dw4st_0d6a813e-46a1-4727-8eeb-362e745cd45d/download-server/0.log" Jul 30 17:37:39.811947 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:39.811919 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q"] Jul 30 17:37:39.812987 ip-10-0-130-254 kubenswrapper[2576]: W0730 17:37:39.812961 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f192353_fbca_4aea_a7e8_630edeb1c550.slice/crio-947c824095613662003b860f5710f9b3bd2ae061b117a35afa30157812c6e527 WatchSource:0}: Error finding container 947c824095613662003b860f5710f9b3bd2ae061b117a35afa30157812c6e527: Status 404 returned error can't find the container with id 947c824095613662003b860f5710f9b3bd2ae061b117a35afa30157812c6e527 Jul 30 17:37:40.289684 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:40.289644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" event={"ID":"9f192353-fbca-4aea-a7e8-630edeb1c550","Type":"ContainerStarted","Data":"b91e029b08dc393b52b98bc2bdd0a541f6d6c0498b5944030ba8c98f89b59281"} Jul 30 17:37:40.289890 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:40.289708 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" event={"ID":"9f192353-fbca-4aea-a7e8-630edeb1c550","Type":"ContainerStarted","Data":"947c824095613662003b860f5710f9b3bd2ae061b117a35afa30157812c6e527"} Jul 30 17:37:40.289890 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:40.289768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:40.309920 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:40.309849 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" podStartSLOduration=1.309826167 podStartE2EDuration="1.309826167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:37:40.307627598 +0000 UTC m=+2229.980825077" watchObservedRunningTime="2026-07-30 17:37:40.309826167 +0000 UTC m=+2229.983023645" Jul 30 17:37:40.337607 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:40.337579 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-mrf7h_07b13ac7-3780-4f15-892e-49fff37fec07/volume-data-source-validator/0.log" Jul 30 17:37:41.235086 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:41.235063 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2twfz_643e3b9d-d9a0-401d-bc08-0ba5554bbba8/dns/0.log" Jul 30 17:37:41.259559 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:41.259530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2twfz_643e3b9d-d9a0-401d-bc08-0ba5554bbba8/kube-rbac-proxy/0.log" Jul 30 17:37:41.441562 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:41.441531 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfrvd_caf62b48-154a-4b0c-941b-506218889e2d/dns-node-resolver/0.log" Jul 30 17:37:41.886285 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:41.886253 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6875b55898-drb6h_8e0e83a3-3106-4899-9221-b31add3218ce/registry/0.log" Jul 30 17:37:41.962581 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:41.962543 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v4kvx_78e6945a-53cb-4b71-af2e-b91cdfdbcb90/node-ca/0.log" Jul 30 17:37:43.051070 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:43.051036 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-446v7_d1e2f7dc-a057-4466-8c27-b93d291eff81/serve-healthcheck-canary/0.log" Jul 30 17:37:43.546022 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:43.545989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-c795x_3ec7ff92-af94-4d0f-ace3-5d7adef84338/insights-operator/0.log" Jul 30 17:37:43.546022 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:43.545999 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-c795x_3ec7ff92-af94-4d0f-ace3-5d7adef84338/insights-operator/1.log" Jul 30 17:37:43.589208 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:43.589178 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ppfb_6eaca9fc-0810-4b49-9e7b-b887e6407fc1/kube-rbac-proxy/0.log" Jul 30 17:37:43.619671 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:43.619645 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ppfb_6eaca9fc-0810-4b49-9e7b-b887e6407fc1/exporter/0.log" Jul 30 17:37:43.651889 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:43.651860 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ppfb_6eaca9fc-0810-4b49-9e7b-b887e6407fc1/extractor/0.log" Jul 30 17:37:46.304804 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:46.304774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-crd4q" Jul 30 17:37:46.784998 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:46.784970 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nvnpg_6b147fe4-9a73-46a3-b456-bd68f9e8a43c/s3-init/0.log" Jul 30 17:37:46.802594 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:46.802571 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nvnpg_6b147fe4-9a73-46a3-b456-bd68f9e8a43c/download-hf-model/0.log" Jul 30 17:37:51.538060 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:51.537953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl_2633ff8f-337d-48db-ace0-2316e62d08b5/kube-storage-version-migrator-operator/1.log" Jul 30 17:37:51.539007 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:51.538988 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-lddkl_2633ff8f-337d-48db-ace0-2316e62d08b5/kube-storage-version-migrator-operator/0.log" Jul 30 17:37:52.726135 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:52.726053 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzck8_db4dcb51-1f61-41d0-9f81-d5b71f4e1c49/kube-multus-additional-cni-plugins/0.log" Jul 30 17:37:52.752997 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:52.752970 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzck8_db4dcb51-1f61-41d0-9f81-d5b71f4e1c49/egress-router-binary-copy/0.log" Jul 30 17:37:52.776225 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:52.776198 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzck8_db4dcb51-1f61-41d0-9f81-d5b71f4e1c49/cni-plugins/0.log" Jul 30 17:37:52.801722 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:52.801697 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzck8_db4dcb51-1f61-41d0-9f81-d5b71f4e1c49/bond-cni-plugin/0.log" Jul 30 17:37:52.825421 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:52.825364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzck8_db4dcb51-1f61-41d0-9f81-d5b71f4e1c49/routeoverride-cni/0.log" Jul 30 17:37:52.853729 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:52.853697 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzck8_db4dcb51-1f61-41d0-9f81-d5b71f4e1c49/whereabouts-cni-bincopy/0.log" Jul 30 17:37:52.877224 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:52.877193 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lzck8_db4dcb51-1f61-41d0-9f81-d5b71f4e1c49/whereabouts-cni/0.log" Jul 30 17:37:53.100970 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:53.100894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qxth4_063d41b9-4280-4bfa-9eed-4ce6ef4084d3/kube-multus/0.log" Jul 30 17:37:53.182538 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:53.182481 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jt82f_6bd1a463-eb1a-4afb-aba9-ea8294291672/network-metrics-daemon/0.log" Jul 30 17:37:53.204570 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:53.204542 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jt82f_6bd1a463-eb1a-4afb-aba9-ea8294291672/kube-rbac-proxy/0.log" Jul 30 17:37:54.302106 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.302064 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/ovn-controller/0.log" Jul 30 17:37:54.324124 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.324099 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/ovn-acl-logging/0.log" Jul 30 17:37:54.353390 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.353356 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/kube-rbac-proxy-node/0.log" Jul 30 17:37:54.378601 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.378554 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/kube-rbac-proxy-ovn-metrics/0.log" Jul 30 17:37:54.402362 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.402340 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/northd/0.log" Jul 30 17:37:54.431363 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.431342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/nbdb/0.log" Jul 30 17:37:54.457836 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.457817 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/sbdb/0.log" Jul 30 17:37:54.502866 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:54.502841 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p8znm_85ef8995-7981-46e0-b9b0-b0b7b4234643/ovnkube-controller/0.log" Jul 30 17:37:55.795411 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:55.795369 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-8bkcp_dbb3afa1-388f-48d4-a3fe-3565261c075f/check-endpoints/0.log" Jul 30 17:37:55.847963 ip-10-0-130-254 kubenswrapper[2576]: I0730 17:37:55.847937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rsvsz_47edcc66-8e41-41b6-9bb3-c457deebbda9/network-check-target-container/0.log"