Jul 15 10:44:12.258859 ip-10-0-130-140 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 10:44:12.258874 ip-10-0-130-140 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 10:44:12.258884 ip-10-0-130-140 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 10:44:12.259233 ip-10-0-130-140 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 10:44:22.444453 ip-10-0-130-140 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 10:44:22.444471 ip-10-0-130-140 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e2d6dae228114279a5006d1695b59293 -- Jul 15 10:46:22.091064 ip-10-0-130-140 systemd[1]: Starting Kubernetes Kubelet... Jul 15 10:46:22.566168 ip-10-0-130-140 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 10:46:22.566168 ip-10-0-130-140 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 10:46:22.566168 ip-10-0-130-140 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 10:46:22.566168 ip-10-0-130-140 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 10:46:22.566168 ip-10-0-130-140 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 10:46:22.568252 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.568081 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 10:46:22.573806 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573780 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:46:22.573806 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573805 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573810 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573814 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573818 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573821 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573824 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573827 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573830 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573833 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573838 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573841 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573843 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573846 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573849 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573852 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573855 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573858 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573861 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573865 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573868 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:46:22.573889 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573872 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573875 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573878 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573881 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573884 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573887 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573890 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573893 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573896 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573899 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573902 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573905 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573908 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573911 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573915 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573917 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573920 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573923 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573926 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573928 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:46:22.574379 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573931 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573934 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573937 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573939 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573942 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573945 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573948 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573951 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573953 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573958 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573963 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573967 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573970 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573974 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573978 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573984 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573988 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573991 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:46:22.574907 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573994 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.573997 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574000 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574003 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574006 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574009 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574011 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574014 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574017 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574020 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574023 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574026 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574029 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574032 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574035 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574037 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574040 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574043 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574046 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574049 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:46:22.575407 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574052 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574055 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574058 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574061 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574064 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574067 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574070 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574073 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574077 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574080 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574083 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574086 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574088 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574091 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574096 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574099 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574102 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574105 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574108 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574111 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:46:22.575931 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574949 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574956 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574959 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574962 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574966 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574968 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574972 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574975 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574978 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574981 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574983 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574986 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574989 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574992 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574995 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.574998 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575001 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575003 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575006 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575010 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:46:22.576479 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575015 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575020 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575023 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575026 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575028 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575032 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575036 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575040 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575044 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575048 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575052 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575055 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575058 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575062 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575065 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575068 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575071 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575074 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575077 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:46:22.577024 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575080 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575083 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575086 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575089 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575092 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575095 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575097 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575100 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575102 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575105 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575107 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575111 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575114 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575118 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575121 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575125 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575127 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575130 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575133 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575135 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:46:22.577556 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575138 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575141 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575143 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575146 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575152 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575155 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575159 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575162 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575165 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575167 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575170 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575173 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575175 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575178 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575181 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575184 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575187 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575189 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575192 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:46:22.578100 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575195 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575197 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575200 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575203 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575206 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575209 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575212 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575215 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575218 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575221 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575223 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575226 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575229 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575232 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575234 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575237 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575240 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575242 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575245 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575247 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:46:22.578601 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575250 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575752 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575760 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575764 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575769 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575772 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575776 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575779 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575782 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575784 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575787 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575790 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575793 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575795 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575798 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575801 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575804 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575807 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575810 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:46:22.579157 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575813 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575816 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575820 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575823 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575826 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575829 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575831 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575834 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575837 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575840 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575842 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575845 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575848 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575850 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575854 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575856 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575859 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575862 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575865 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575867 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:46:22.579643 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575870 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575873 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575876 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575882 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575885 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575888 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575890 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575893 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575897 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575900 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575903 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575907 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575911 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575914 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575917 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575921 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575924 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575927 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575930 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575934 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:46:22.580188 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575937 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575941 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575944 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575947 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575950 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575953 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575956 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575959 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575962 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575965 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575968 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575970 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575973 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575976 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575979 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575983 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575986 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575989 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575992 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575995 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:46:22.580728 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.575997 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576000 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576003 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576006 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576009 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576013 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576016 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576018 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576022 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576026 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576030 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576033 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576036 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576039 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576042 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576045 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576050 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576053 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576056 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:46:22.581259 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576058 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576061 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576138 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576146 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576153 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576157 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576161 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576165 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576172 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576176 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576180 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576183 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576187 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576191 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576194 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576197 2580 flags.go:64] FLAG: --cgroup-root="" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576201 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576204 2580 flags.go:64] FLAG: --client-ca-file="" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576207 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576210 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576214 2580 flags.go:64] FLAG: --cluster-domain="" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576218 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576222 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576225 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 10:46:22.581780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576229 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576233 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576237 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576240 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576244 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576249 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576252 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576256 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576259 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576263 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576268 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576271 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576275 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576278 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576281 2580 flags.go:64] FLAG: --enable-server="true" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576284 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576290 2580 flags.go:64] FLAG: --event-burst="100" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576294 2580 flags.go:64] FLAG: --event-qps="50" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576297 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576300 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576304 2580 flags.go:64] FLAG: --eviction-hard="" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576308 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576311 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576314 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576318 2580 flags.go:64] FLAG: --eviction-soft="" Jul 15 10:46:22.582399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576321 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576324 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576328 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576331 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576335 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576338 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576341 2580 flags.go:64] FLAG: --feature-gates="" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576345 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576349 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576352 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576356 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576359 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576364 2580 flags.go:64] FLAG: --help="false" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576368 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576371 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576374 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576377 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576380 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576384 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576387 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576391 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576394 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576397 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 10:46:22.583105 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576401 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576405 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576408 2580 flags.go:64] FLAG: --kube-reserved="" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576411 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576414 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576418 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576421 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576424 2580 flags.go:64] FLAG: --lock-file="" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576427 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576430 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576433 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576439 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576442 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576446 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576449 2580 flags.go:64] FLAG: --logging-format="text" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576452 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576456 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576459 2580 flags.go:64] FLAG: --manifest-url="" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576462 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576466 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576470 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576479 2580 flags.go:64] FLAG: --max-pods="110" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576483 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576486 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576489 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 10:46:22.583749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576493 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576496 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576499 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576502 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576511 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576515 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576518 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576523 2580 flags.go:64] FLAG: --pod-cidr="" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576526 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576532 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576535 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576539 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576542 2580 flags.go:64] FLAG: --port="10250" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576545 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576548 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0682873edc6adae76" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576552 2580 flags.go:64] FLAG: --qos-reserved="" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576555 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576558 2580 flags.go:64] FLAG: --register-node="true" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576561 2580 flags.go:64] FLAG: --register-with-taints="" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576565 2580 flags.go:64] FLAG: --registry-burst="10" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576568 2580 flags.go:64] FLAG: --registry-qps="5" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576571 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576574 2580 flags.go:64] FLAG: --reserved-memory="" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576578 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576582 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 10:46:22.584425 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576585 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576588 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576591 2580 flags.go:64] FLAG: --runonce="false" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576597 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576600 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576603 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576606 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576609 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576613 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576616 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576619 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576622 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576625 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576628 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576631 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576635 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576638 2580 flags.go:64] FLAG: --system-cgroups="" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576641 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576664 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576667 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576670 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576674 2580 flags.go:64] FLAG: --tls-min-version="" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576677 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576681 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576684 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 10:46:22.585100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576687 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576690 2580 flags.go:64] FLAG: --v="2" Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576695 2580 flags.go:64] FLAG: --version="false" Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576699 2580 flags.go:64] FLAG: --vmodule="" Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576703 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.576707 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576842 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576848 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576852 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576856 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576860 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576863 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576867 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576870 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576872 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576875 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576878 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576880 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576883 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576886 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576888 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:46:22.585777 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576891 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576895 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576897 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576900 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576903 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576906 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576909 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576911 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576914 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576917 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576919 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576922 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576925 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576927 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576930 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576933 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576935 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576938 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576941 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576944 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:46:22.586335 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576946 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576950 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576953 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576956 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576959 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576961 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576964 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576986 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576990 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576994 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576997 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.576999 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577003 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577006 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577008 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577011 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577014 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577016 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577019 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:46:22.587023 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577022 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577025 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577028 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577031 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577034 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577037 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577040 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577042 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577045 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577048 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577050 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577053 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577056 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577059 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577063 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577067 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577071 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577074 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577077 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577080 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:46:22.587587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577083 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577086 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577088 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577091 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577094 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577096 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577101 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577103 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577107 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577111 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577115 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577118 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577120 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577124 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577127 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577130 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577132 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577136 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577139 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577141 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:46:22.588181 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577144 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577147 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577150 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577152 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.577155 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.577161 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.587890 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.587913 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.587942 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.587953 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588030 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588037 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588043 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588048 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:46:22.588714 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588052 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588057 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588061 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588064 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588067 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588071 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588074 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588077 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588080 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588083 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588086 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588089 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588092 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588097 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588101 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588106 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588111 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588116 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588126 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:46:22.589085 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588129 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588132 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588135 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588138 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588141 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588144 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588148 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588151 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588154 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588157 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588160 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588162 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588165 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588168 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588171 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588174 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588177 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588179 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588182 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588187 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:46:22.589587 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588191 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588196 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588200 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588206 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588209 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588213 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588216 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588219 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588222 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588225 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588227 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588230 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588233 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588236 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588239 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588242 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588245 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588247 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588251 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588254 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:46:22.590129 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588257 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588259 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588262 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588265 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588268 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588270 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588276 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588281 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588287 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588292 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588296 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588299 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588302 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588305 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588308 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588311 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588315 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588318 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588320 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:46:22.590870 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588323 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588327 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588329 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588332 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588335 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588338 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588341 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588343 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588346 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588349 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588357 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588361 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588363 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588367 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588370 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588372 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:46:22.591369 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588375 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.588381 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588529 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588535 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588540 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588544 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588549 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588554 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588559 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588563 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588567 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588570 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588573 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588576 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588580 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 10:46:22.591805 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588583 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588586 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588589 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588592 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588595 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588598 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588600 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588603 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588606 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588609 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588611 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588614 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588617 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588620 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588622 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588625 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588628 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588631 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588633 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588636 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 10:46:22.592204 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588639 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588641 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588644 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588647 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588664 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588667 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588671 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588675 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588679 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588682 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588686 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588689 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588693 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588697 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588700 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588702 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588705 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588708 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588711 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 10:46:22.592735 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588714 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588717 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588720 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588723 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588726 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588729 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588733 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588736 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588738 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588741 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588744 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588747 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588750 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588753 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588756 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588758 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588761 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588764 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588767 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588770 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 10:46:22.593255 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588773 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588776 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588781 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588785 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588790 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588794 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588802 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588806 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588810 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588813 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588817 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588820 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588822 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588825 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588828 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588830 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588833 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588837 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588839 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 10:46:22.593842 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588842 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588845 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588848 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588851 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588854 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588857 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588860 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:22.588863 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.588868 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.589010 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 10:46:22.594339 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.593390 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 10:46:22.595048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.595033 2580 server.go:1019] "Starting client certificate rotation" Jul 15 10:46:22.595186 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.595163 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 10:46:22.595259 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.595223 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 10:46:22.597829 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.597807 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 10:46:22.599125 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.599099 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 10:46:22.607804 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.607780 2580 log.go:25] "Validated CRI v1 runtime API" Jul 15 10:46:22.610313 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.610293 2580 log.go:25] "Validated CRI v1 image API" Jul 15 10:46:22.611257 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.611236 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 10:46:22.614575 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.614540 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad1d71ad-0160-40ba-9bb1-2cfeed64ae16:/dev/nvme0n1p3 d3c049fb-be62-425d-ad9f-6166c80a0ecc:/dev/nvme0n1p4] Jul 15 10:46:22.614674 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.614575 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 10:46:22.621482 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.621354 2580 manager.go:216] Machine: {Timestamp:2026-07-15 10:46:22.619824558 +0000 UTC m=+0.396275811 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098539 MemoryCapacity:33164505088 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c3ad973a349788e19653d2c2d9a0e SystemUUID:ec2c3ad9-73a3-4978-8e19-653d2c2d9a0e BootID:e2d6dae2-2811-4279-a500-6d1695b59293 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582254592 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e6:1e:74:99:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e6:1e:74:99:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:e5:19:8a:ea:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164505088 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 10:46:22.621482 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.621471 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 10:46:22.621618 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.621574 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 10:46:22.625599 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.625562 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 10:46:22.625780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.625605 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 10:46:22.626448 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.626436 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 10:46:22.626483 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.626451 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 10:46:22.626483 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.626459 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 10:46:22.627095 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.627082 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 10:46:22.627945 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.627933 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 10:46:22.630818 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.630802 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 15 10:46:22.630956 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.630946 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 10:46:22.631627 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.631617 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 15 10:46:22.631675 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.631634 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 10:46:22.631675 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.631666 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 10:46:22.631739 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.631678 2580 kubelet.go:398] "Adding apiserver pod source" Jul 15 10:46:22.631739 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.631689 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 10:46:22.632587 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.632544 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 10:46:22.632764 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.632748 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 10:46:22.635814 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.635791 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 10:46:22.637608 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.637588 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 10:46:22.637723 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.637631 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 10:46:22.638519 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638503 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 10:46:22.638591 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638525 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 10:46:22.638591 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638535 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 10:46:22.638591 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638544 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 10:46:22.638591 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638559 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 10:46:22.638591 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638569 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 10:46:22.638811 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638685 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 10:46:22.638811 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638697 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 10:46:22.638811 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638709 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 10:46:22.638811 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638718 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 10:46:22.638811 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638733 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 10:46:22.638811 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.638749 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 10:46:22.639149 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.639129 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 10:46:22.639574 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.639562 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 10:46:22.639626 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.639578 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 10:46:22.643172 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.643144 2580 server.go:1268] "Started kubelet" Jul 15 10:46:22.643290 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.643258 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 10:46:22.643799 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.643233 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 10:46:22.643799 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.643610 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 10:46:22.643961 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.643939 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 10:46:22.644008 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.643981 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 10:46:22.644047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.644013 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 10:46:22.644206 ip-10-0-130-140 systemd[1]: Started Kubernetes Kubelet. Jul 15 10:46:22.645807 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.645792 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 15 10:46:22.647716 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.647696 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 10:46:22.649606 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.649571 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 10:46:22.650151 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.649087 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-140.ec2.internal.18c270c266d18b12 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-140.ec2.internal,UID:ip-10-0-130-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-140.ec2.internal,},FirstTimestamp:2026-07-15 10:46:22.643112722 +0000 UTC m=+0.419563954,LastTimestamp:2026-07-15 10:46:22.643112722 +0000 UTC m=+0.419563954,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-140.ec2.internal,}" Jul 15 10:46:22.651572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.651557 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 10:46:22.651631 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.651596 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 10:46:22.651697 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.651659 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:22.652189 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652169 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 10:46:22.652285 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652224 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 10:46:22.652285 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652240 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 10:46:22.652390 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652310 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 15 10:46:22.652390 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652320 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 15 10:46:22.652477 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652441 2580 factory.go:55] Registering systemd factory Jul 15 10:46:22.652477 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652473 2580 factory.go:223] Registration of the systemd container factory successfully Jul 15 10:46:22.652759 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652739 2580 factory.go:153] Registering CRI-O factory Jul 15 10:46:22.652759 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652761 2580 factory.go:223] Registration of the crio container factory successfully Jul 15 10:46:22.652967 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652814 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 10:46:22.652967 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652840 2580 factory.go:103] Registering Raw factory Jul 15 10:46:22.652967 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.652855 2580 manager.go:1196] Started watching for new ooms in manager Jul 15 10:46:22.653252 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.653239 2580 manager.go:349] Starting recovery of all containers Jul 15 10:46:22.664181 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.663427 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pn2hd" Jul 15 10:46:22.665661 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.665620 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 10:46:22.665661 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.665623 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 10:46:22.666370 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.666348 2580 manager.go:354] Recovery completed Jul 15 10:46:22.671914 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.671898 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:46:22.674170 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.674149 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:46:22.674241 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.674187 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:46:22.674241 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.674198 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:46:22.674835 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.674819 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 10:46:22.674835 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.674832 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:46:22.674971 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.674834 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 10:46:22.674971 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.674883 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 15 10:46:22.675613 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.675582 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:46:22.675730 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.675625 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:46:22.675730 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.675640 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:46:22.676159 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.676146 2580 policy_none.go:49] "None policy: Start" Jul 15 10:46:22.676199 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.676166 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 10:46:22.676199 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.676178 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 10:46:22.677534 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.677522 2580 policy_none.go:47] "Start" Jul 15 10:46:22.682293 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.682276 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pn2hd" Jul 15 10:46:22.686058 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.685958 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-140.ec2.internal.18c270c268ab7192 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-140.ec2.internal,UID:ip-10-0-130-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-140.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-140.ec2.internal,},FirstTimestamp:2026-07-15 10:46:22.674170258 +0000 UTC m=+0.450621488,LastTimestamp:2026-07-15 10:46:22.674170258 +0000 UTC m=+0.450621488,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-140.ec2.internal,}" Jul 15 10:46:22.716610 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.716423 2580 manager.go:337] "Starting Device Plugin manager" Jul 15 10:46:22.716776 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.716636 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 10:46:22.716776 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.716668 2580 server.go:87] "Starting device plugin registration server" Jul 15 10:46:22.717055 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.717041 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 10:46:22.717127 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.717059 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 10:46:22.717186 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.717158 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 10:46:22.717275 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.717250 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 10:46:22.717275 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.717261 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 10:46:22.719284 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.719260 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 10:46:22.719380 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.719322 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:22.817567 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.817480 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:46:22.817567 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.817483 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 10:46:22.818825 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.818487 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:46:22.818825 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.818525 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:46:22.818825 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.818542 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:46:22.818825 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.818601 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.819076 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.819056 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 10:46:22.819133 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.819100 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 10:46:22.819133 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.819128 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 10:46:22.819234 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.819175 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 10:46:22.823749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.823730 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:46:22.827954 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.827934 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.828016 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.827968 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-140.ec2.internal\": node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:22.888290 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.888256 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:22.920689 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.920635 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal"] Jul 15 10:46:22.920781 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.920743 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:46:22.921818 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.921801 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:46:22.921886 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.921832 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:46:22.921886 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.921842 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:46:22.923383 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.923369 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:46:22.923515 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.923497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.923570 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.923560 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:46:22.924222 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.924198 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:46:22.924345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.924229 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:46:22.924345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.924239 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:46:22.924345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.924252 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:46:22.924345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.924257 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:46:22.924345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.924265 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:46:22.925621 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.925605 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.925692 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.925632 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 10:46:22.926536 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.926520 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 10:46:22.926630 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.926549 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 10:46:22.926630 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.926562 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 10:46:22.943450 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.943425 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-140.ec2.internal\" not found" node="ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.947542 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.947524 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-140.ec2.internal\" not found" node="ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.953934 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.953907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f592e6633d80c688b1a3245771d9573-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal\" (UID: \"6f592e6633d80c688b1a3245771d9573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.954022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.953941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f592e6633d80c688b1a3245771d9573-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal\" (UID: \"6f592e6633d80c688b1a3245771d9573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.954022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:22.953962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04c4f391b82cdea4e8f49294bf51b41d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-140.ec2.internal\" (UID: \"04c4f391b82cdea4e8f49294bf51b41d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" Jul 15 10:46:22.988561 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:22.988532 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:23.054721 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.054686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f592e6633d80c688b1a3245771d9573-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal\" (UID: \"6f592e6633d80c688b1a3245771d9573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.054908 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.054733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f592e6633d80c688b1a3245771d9573-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal\" (UID: \"6f592e6633d80c688b1a3245771d9573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.054908 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.054758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04c4f391b82cdea4e8f49294bf51b41d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-140.ec2.internal\" (UID: \"04c4f391b82cdea4e8f49294bf51b41d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.054908 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.054766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f592e6633d80c688b1a3245771d9573-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal\" (UID: \"6f592e6633d80c688b1a3245771d9573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.054908 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.054796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f592e6633d80c688b1a3245771d9573-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal\" (UID: \"6f592e6633d80c688b1a3245771d9573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.054908 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.054814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04c4f391b82cdea4e8f49294bf51b41d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-140.ec2.internal\" (UID: \"04c4f391b82cdea4e8f49294bf51b41d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.090514 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.090427 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:23.192508 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.192455 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:23.250054 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.250007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.254618 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.254350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.292787 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.292742 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:23.395780 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.395676 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:23.497235 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.497195 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-140.ec2.internal\" not found" Jul 15 10:46:23.583282 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.583251 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:46:23.594185 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.594151 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 10:46:23.594339 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.594317 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 10:46:23.594401 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.594315 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 10:46:23.634643 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.634605 2580 apiserver.go:52] "Watching apiserver" Jul 15 10:46:23.645387 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.645362 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:46:23.652156 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.652104 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 10:46:23.652232 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.652166 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 10:46:23.652614 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.652594 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.659866 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.659822 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mczbg","openshift-ovn-kubernetes/ovnkube-node-22ll8","kube-system/global-pull-secret-syncer-d8q55","openshift-cluster-node-tuning-operator/tuned-4tvsj","openshift-dns/node-resolver-9npvb","openshift-image-registry/node-ca-j98bg","openshift-network-diagnostics/network-check-target-2fw2b","openshift-network-operator/iptables-alerter-mqts8","kube-system/konnectivity-agent-s8dpc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk","openshift-multus/multus-additional-cni-plugins-gwd87","openshift-multus/multus-slrhd"] Jul 15 10:46:23.662437 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.662410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:23.662548 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.662498 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:23.663828 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.663808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.665625 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.665599 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.665770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.665746 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:23.666672 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.666645 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 10:46:23.666786 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.666769 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 10:46:23.666954 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.666931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.667218 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.667120 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 10:46:23.667218 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.667150 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 10:46:23.667218 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.667122 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 10:46:23.667444 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.667417 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 10:46:23.667548 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.667528 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rsxk9\"" Jul 15 10:46:23.668398 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.668372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.669659 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.669625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.670091 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.670075 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 10:46:23.670271 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.670257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-82xcj\"" Jul 15 10:46:23.670521 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.670509 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 10:46:23.670985 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.670971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x2qjk\"" Jul 15 10:46:23.671031 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.671012 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 10:46:23.671743 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.671730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:23.671814 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.671798 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:23.672578 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.672562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h42xm\"" Jul 15 10:46:23.672634 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.672613 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 10:46:23.672720 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.672655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 10:46:23.672720 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.672656 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 10:46:23.672720 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.672619 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 10:46:23.673141 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.673126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.674491 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.674472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:23.676037 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.676020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.676970 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.676954 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 10:46:23.677075 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677057 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" Jul 15 10:46:23.677237 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677220 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 10:46:23.677396 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677233 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 10:46:23.677396 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677388 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 10:46:23.677561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677424 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 10:46:23.677561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677520 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 10:46:23.677690 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.677933 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.677912 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r5p2x\"" Jul 15 10:46:23.678302 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.678280 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rqzxb\"" Jul 15 10:46:23.679359 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.679339 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal"] Jul 15 10:46:23.679441 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.679344 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.681714 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.681694 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 10:46:23.681836 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.681816 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2v2vl\"" Jul 15 10:46:23.681893 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.681881 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 10:46:23.681942 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.681896 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 10:46:23.681942 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.681901 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 10:46:23.684661 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.684631 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 10:46:23.684821 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.684803 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 10:46:23.684922 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.684897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pp6jw\"" Jul 15 10:46:23.684922 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.684904 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 10:46:23.685031 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.684950 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 10:46:23.685079 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.685035 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 10:46:23.685079 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.685075 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 10:46:23.685164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.685096 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5m7ms\"" Jul 15 10:46:23.686402 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.686370 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 10:41:23 +0000 UTC" deadline="2028-03-14 22:31:49.430583237 +0000 UTC" Jul 15 10:46:23.686402 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.686400 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14603h45m25.74418577s" Jul 15 10:46:23.693101 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.693083 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 10:46:23.693755 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.693738 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal"] Jul 15 10:46:23.712808 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.712767 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7ssd" Jul 15 10:46:23.723267 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.723243 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7ssd" Jul 15 10:46:23.753238 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.753211 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 10:46:23.759644 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm74b\" (UniqueName: \"kubernetes.io/projected/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-kube-api-access-mm74b\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.759801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8kc\" (UniqueName: \"kubernetes.io/projected/40d807c3-969a-4305-a000-5eb13577cd18-kube-api-access-cr8kc\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.759801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-system-cni-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.759801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-cni-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.759801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-cni-multus\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.759801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfn6\" (UniqueName: \"kubernetes.io/projected/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-kube-api-access-pwfn6\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.759801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fgr\" (UniqueName: \"kubernetes.io/projected/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-kube-api-access-h7fgr\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.759801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6e6667c-5712-4706-943a-572aa76de3d3-etc-tuned\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-device-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-etc-kubernetes\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-sys-fs\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.759959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26139f2-0422-42d7-b310-9cc1693f69ad-cni-binary-copy\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.760064 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-netns\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-conf-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpt7\" (UniqueName: \"kubernetes.io/projected/a6e6667c-5712-4706-943a-572aa76de3d3-kube-api-access-4zpt7\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-daemon-config\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-systemd\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-etc-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-cni-netd\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f7beef7-c62f-468d-b02c-2a06d2322773-kubelet-config\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysconfig\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-log-socket\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovnkube-config\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-host-slash\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.760408 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-cni-bin\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovn-node-metrics-cert\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f7beef7-c62f-468d-b02c-2a06d2322773-dbus\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstpc\" (UniqueName: \"kubernetes.io/projected/41649f74-e757-4ee4-9df0-a63755eb1b0c-kube-api-access-xstpc\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-iptables-alerter-script\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-var-lib-kubelet\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-kubelet\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovnkube-script-lib\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9lf\" (UniqueName: \"kubernetes.io/projected/f8874c1d-1f5f-4529-ac5b-bcec15627398-kube-api-access-vh9lf\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-run\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-sys\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec45a64a-a53a-4025-93dd-938828aa0269-agent-certs\") pod \"konnectivity-agent-s8dpc\" (UID: \"ec45a64a-a53a-4025-93dd-938828aa0269\") " pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-multus-certs\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-modprobe-d\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6e6667c-5712-4706-943a-572aa76de3d3-tmp\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.761083 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec45a64a-a53a-4025-93dd-938828aa0269-konnectivity-ca\") pod \"konnectivity-agent-s8dpc\" (UID: \"ec45a64a-a53a-4025-93dd-938828aa0269\") " pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-socket-dir-parent\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.760979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-systemd-units\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-node-log\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-socket-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-os-release\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-cnibin\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-slash\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysctl-conf\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-k8s-cni-cncf-io\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-hostroot\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-registration-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-ovn\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.761793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-hosts-file\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-tmp-dir\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41649f74-e757-4ee4-9df0-a63755eb1b0c-serviceca\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnbq\" (UniqueName: \"kubernetes.io/projected/f26139f2-0422-42d7-b310-9cc1693f69ad-kube-api-access-bgnbq\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-host\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-kubernetes\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-lib-modules\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-cni-bin\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41649f74-e757-4ee4-9df0-a63755eb1b0c-host\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-system-cni-dir\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-run-netns\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-var-lib-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-systemd\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-cnibin\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.762329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-os-release\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.763178 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5bs\" (UniqueName: \"kubernetes.io/projected/0119351c-bc27-4885-b458-ba76a4390657-kube-api-access-4n5bs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:23.763178 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-kubelet\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.763178 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-env-overrides\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.763178 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.761921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysctl-d\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.862261 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-sys-fs\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.862261 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26139f2-0422-42d7-b310-9cc1693f69ad-cni-binary-copy\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-netns\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-sys-fs\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-conf-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862438 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-netns\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpt7\" (UniqueName: \"kubernetes.io/projected/a6e6667c-5712-4706-943a-572aa76de3d3-kube-api-access-4zpt7\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-conf-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-daemon-config\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.862572 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-systemd\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-etc-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-cni-netd\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f7beef7-c62f-468d-b02c-2a06d2322773-kubelet-config\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-systemd\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysconfig\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862717 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-etc-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f7beef7-c62f-468d-b02c-2a06d2322773-kubelet-config\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-log-socket\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862786 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysconfig\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-cni-netd\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovnkube-config\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-log-socket\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-host-slash\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-run-ovn-kubernetes\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863047 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-cni-bin\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.862998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-host-slash\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovn-node-metrics-cert\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863035 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26139f2-0422-42d7-b310-9cc1693f69ad-cni-binary-copy\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863043 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-cni-bin\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f7beef7-c62f-468d-b02c-2a06d2322773-dbus\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863037 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-daemon-config\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstpc\" (UniqueName: \"kubernetes.io/projected/41649f74-e757-4ee4-9df0-a63755eb1b0c-kube-api-access-xstpc\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-iptables-alerter-script\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-var-lib-kubelet\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-kubelet\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-kubelet\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-var-lib-kubelet\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovnkube-script-lib\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f7beef7-c62f-468d-b02c-2a06d2322773-dbus\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.863965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9lf\" (UniqueName: \"kubernetes.io/projected/f8874c1d-1f5f-4529-ac5b-bcec15627398-kube-api-access-vh9lf\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-run\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-sys\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec45a64a-a53a-4025-93dd-938828aa0269-agent-certs\") pod \"konnectivity-agent-s8dpc\" (UID: \"ec45a64a-a53a-4025-93dd-938828aa0269\") " pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863410 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-multus-certs\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovnkube-config\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-multus-certs\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-run\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-modprobe-d\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6e6667c-5712-4706-943a-572aa76de3d3-tmp\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-sys\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec45a64a-a53a-4025-93dd-938828aa0269-konnectivity-ca\") pod \"konnectivity-agent-s8dpc\" (UID: \"ec45a64a-a53a-4025-93dd-938828aa0269\") " pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-socket-dir-parent\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-iptables-alerter-script\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-systemd-units\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.864788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-node-log\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-modprobe-d\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-socket-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovnkube-script-lib\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-systemd-units\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-socket-dir-parent\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-node-log\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.863943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-os-release\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-socket-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-cnibin\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-slash\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-os-release\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysctl-conf\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-k8s-cni-cncf-io\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-cnibin\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.865782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-hostroot\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-registration-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-ovn\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysctl-conf\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-hosts-file\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec45a64a-a53a-4025-93dd-938828aa0269-konnectivity-ca\") pod \"konnectivity-agent-s8dpc\" (UID: \"ec45a64a-a53a-4025-93dd-938828aa0269\") " pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-tmp-dir\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-slash\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41649f74-e757-4ee4-9df0-a63755eb1b0c-serviceca\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-registration-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnbq\" (UniqueName: \"kubernetes.io/projected/f26139f2-0422-42d7-b310-9cc1693f69ad-kube-api-access-bgnbq\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-hostroot\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-hosts-file\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-host\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-run-k8s-cni-cncf-io\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-kubernetes\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.866601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.864599 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-lib-modules\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d807c3-969a-4305-a000-5eb13577cd18-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.864713 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret podName:4f7beef7-c62f-468d-b02c-2a06d2322773 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:24.364666441 +0000 UTC m=+2.141117675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret") pod "global-pull-secret-syncer-d8q55" (UID: "4f7beef7-c62f-468d-b02c-2a06d2322773") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-kubernetes\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-host\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-cni-bin\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41649f74-e757-4ee4-9df0-a63755eb1b0c-host\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-lib-modules\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-system-cni-dir\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864896 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-system-cni-dir\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-run-netns\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-tmp-dir\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-cni-bin\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.867488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-var-lib-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41649f74-e757-4ee4-9df0-a63755eb1b0c-host\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-var-lib-openvswitch\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-run-ovn\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.864971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-systemd\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-cnibin\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-run-netns\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-systemd\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-os-release\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-cnibin\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5bs\" (UniqueName: \"kubernetes.io/projected/0119351c-bc27-4885-b458-ba76a4390657-kube-api-access-4n5bs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-kubelet\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-os-release\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-env-overrides\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysctl-d\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41649f74-e757-4ee4-9df0-a63755eb1b0c-serviceca\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d807c3-969a-4305-a000-5eb13577cd18-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8874c1d-1f5f-4529-ac5b-bcec15627398-host-kubelet\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.868338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm74b\" (UniqueName: \"kubernetes.io/projected/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-kube-api-access-mm74b\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6e6667c-5712-4706-943a-572aa76de3d3-etc-sysctl-d\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8kc\" (UniqueName: \"kubernetes.io/projected/40d807c3-969a-4305-a000-5eb13577cd18-kube-api-access-cr8kc\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-system-cni-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-cni-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-cni-multus\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865617 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-system-cni-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-multus-cni-dir\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfn6\" (UniqueName: \"kubernetes.io/projected/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-kube-api-access-pwfn6\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-host-var-lib-cni-multus\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fgr\" (UniqueName: \"kubernetes.io/projected/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-kube-api-access-h7fgr\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6e6667c-5712-4706-943a-572aa76de3d3-etc-tuned\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-device-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-etc-kubernetes\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.865885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:23.869048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.866025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.866207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26139f2-0422-42d7-b310-9cc1693f69ad-etc-kubernetes\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.866224 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-device-dir\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.866341 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.866404 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs podName:0119351c-bc27-4885-b458-ba76a4390657 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:24.366385314 +0000 UTC m=+2.142836543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs") pod "network-metrics-daemon-mczbg" (UID: "0119351c-bc27-4885-b458-ba76a4390657") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.866500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8874c1d-1f5f-4529-ac5b-bcec15627398-env-overrides\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.868023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6e6667c-5712-4706-943a-572aa76de3d3-tmp\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.868169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8874c1d-1f5f-4529-ac5b-bcec15627398-ovn-node-metrics-cert\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.868313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec45a64a-a53a-4025-93dd-938828aa0269-agent-certs\") pod \"konnectivity-agent-s8dpc\" (UID: \"ec45a64a-a53a-4025-93dd-938828aa0269\") " pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:23.869561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.868556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6e6667c-5712-4706-943a-572aa76de3d3-etc-tuned\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.876712 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.876600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpt7\" (UniqueName: \"kubernetes.io/projected/a6e6667c-5712-4706-943a-572aa76de3d3-kube-api-access-4zpt7\") pod \"tuned-4tvsj\" (UID: \"a6e6667c-5712-4706-943a-572aa76de3d3\") " pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:23.879525 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.879485 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8kc\" (UniqueName: \"kubernetes.io/projected/40d807c3-969a-4305-a000-5eb13577cd18-kube-api-access-cr8kc\") pod \"multus-additional-cni-plugins-gwd87\" (UID: \"40d807c3-969a-4305-a000-5eb13577cd18\") " pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:23.879697 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.879670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fgr\" (UniqueName: \"kubernetes.io/projected/8c5d1637-944f-41b0-9af0-d93f9b31c9a1-kube-api-access-h7fgr\") pod \"iptables-alerter-mqts8\" (UID: \"8c5d1637-944f-41b0-9af0-d93f9b31c9a1\") " pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:23.879836 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.879814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstpc\" (UniqueName: \"kubernetes.io/projected/41649f74-e757-4ee4-9df0-a63755eb1b0c-kube-api-access-xstpc\") pod \"node-ca-j98bg\" (UID: \"41649f74-e757-4ee4-9df0-a63755eb1b0c\") " pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:23.891855 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.891826 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:46:23.891855 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.891860 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:46:23.892056 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.891874 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf6ft for pod openshift-network-diagnostics/network-check-target-2fw2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:23.892056 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:23.891950 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft podName:3439f5a7-5f74-422e-8089-c97186c19f74 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:24.39192917 +0000 UTC m=+2.168380395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nf6ft" (UniqueName: "kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft") pod "network-check-target-2fw2b" (UID: "3439f5a7-5f74-422e-8089-c97186c19f74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:23.892369 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.892331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnbq\" (UniqueName: \"kubernetes.io/projected/f26139f2-0422-42d7-b310-9cc1693f69ad-kube-api-access-bgnbq\") pod \"multus-slrhd\" (UID: \"f26139f2-0422-42d7-b310-9cc1693f69ad\") " pod="openshift-multus/multus-slrhd" Jul 15 10:46:23.892474 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.892455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm74b\" (UniqueName: \"kubernetes.io/projected/b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a-kube-api-access-mm74b\") pod \"aws-ebs-csi-driver-node-bm6fk\" (UID: \"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:23.892543 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.892506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9lf\" (UniqueName: \"kubernetes.io/projected/f8874c1d-1f5f-4529-ac5b-bcec15627398-kube-api-access-vh9lf\") pod \"ovnkube-node-22ll8\" (UID: \"f8874c1d-1f5f-4529-ac5b-bcec15627398\") " pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.893967 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.893947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfn6\" (UniqueName: \"kubernetes.io/projected/a248225a-5a67-4ad2-9b38-fc3c04ee92ac-kube-api-access-pwfn6\") pod \"node-resolver-9npvb\" (UID: \"a248225a-5a67-4ad2-9b38-fc3c04ee92ac\") " pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:23.894110 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.894087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5bs\" (UniqueName: \"kubernetes.io/projected/0119351c-bc27-4885-b458-ba76a4390657-kube-api-access-4n5bs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:23.899923 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:23.899887 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c4f391b82cdea4e8f49294bf51b41d.slice/crio-fb9c3cdcfe1fcf64b2dc2691ff0f9b733c4f8664acb71fc63d61b3361d777adb WatchSource:0}: Error finding container fb9c3cdcfe1fcf64b2dc2691ff0f9b733c4f8664acb71fc63d61b3361d777adb: Status 404 returned error can't find the container with id fb9c3cdcfe1fcf64b2dc2691ff0f9b733c4f8664acb71fc63d61b3361d777adb Jul 15 10:46:23.907633 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.907609 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 10:46:23.928949 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:23.928918 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f592e6633d80c688b1a3245771d9573.slice/crio-216710675bab9c1e9cb8e83f88044afb17c2a5c505e86e3fb214da408bbfc35b WatchSource:0}: Error finding container 216710675bab9c1e9cb8e83f88044afb17c2a5c505e86e3fb214da408bbfc35b: Status 404 returned error can't find the container with id 216710675bab9c1e9cb8e83f88044afb17c2a5c505e86e3fb214da408bbfc35b Jul 15 10:46:23.991454 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.991411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:23.996230 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:23.996204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" Jul 15 10:46:24.000410 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.000380 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8874c1d_1f5f_4529_ac5b_bcec15627398.slice/crio-18a55351bb135790ddb91e8f2e18f0214fdbb4bf5385db2f6a2f484ead134a66 WatchSource:0}: Error finding container 18a55351bb135790ddb91e8f2e18f0214fdbb4bf5385db2f6a2f484ead134a66: Status 404 returned error can't find the container with id 18a55351bb135790ddb91e8f2e18f0214fdbb4bf5385db2f6a2f484ead134a66 Jul 15 10:46:24.005864 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.005835 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e6667c_5712_4706_943a_572aa76de3d3.slice/crio-ea0ca7d059b6ab91b82147ca0a5c9a4cdd58a2082192216f430883b67e693748 WatchSource:0}: Error finding container ea0ca7d059b6ab91b82147ca0a5c9a4cdd58a2082192216f430883b67e693748: Status 404 returned error can't find the container with id ea0ca7d059b6ab91b82147ca0a5c9a4cdd58a2082192216f430883b67e693748 Jul 15 10:46:24.025109 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.025071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9npvb" Jul 15 10:46:24.034966 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.034934 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda248225a_5a67_4ad2_9b38_fc3c04ee92ac.slice/crio-e8700c42192eb9a56a91b0c62804430e4fdb3892b7c31e51e5265aea0b613f15 WatchSource:0}: Error finding container e8700c42192eb9a56a91b0c62804430e4fdb3892b7c31e51e5265aea0b613f15: Status 404 returned error can't find the container with id e8700c42192eb9a56a91b0c62804430e4fdb3892b7c31e51e5265aea0b613f15 Jul 15 10:46:24.041675 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.041626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j98bg" Jul 15 10:46:24.051449 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.051415 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41649f74_e757_4ee4_9df0_a63755eb1b0c.slice/crio-78322494c3719d88acd51f5cbee1141bf53464fae391815e720570d78aae112d WatchSource:0}: Error finding container 78322494c3719d88acd51f5cbee1141bf53464fae391815e720570d78aae112d: Status 404 returned error can't find the container with id 78322494c3719d88acd51f5cbee1141bf53464fae391815e720570d78aae112d Jul 15 10:46:24.064158 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.064117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mqts8" Jul 15 10:46:24.072767 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.072731 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5d1637_944f_41b0_9af0_d93f9b31c9a1.slice/crio-a33de9d666e9f6af4d4e5486fc34b6db02520d61ea8f4440c3b90a70d0bbdb05 WatchSource:0}: Error finding container a33de9d666e9f6af4d4e5486fc34b6db02520d61ea8f4440c3b90a70d0bbdb05: Status 404 returned error can't find the container with id a33de9d666e9f6af4d4e5486fc34b6db02520d61ea8f4440c3b90a70d0bbdb05 Jul 15 10:46:24.073897 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.073880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:24.081760 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.081731 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec45a64a_a53a_4025_93dd_938828aa0269.slice/crio-3fec637ea5ac5bf19bff2b647c39241959d924e29aa6cfe8b1b29f1ad1ca0864 WatchSource:0}: Error finding container 3fec637ea5ac5bf19bff2b647c39241959d924e29aa6cfe8b1b29f1ad1ca0864: Status 404 returned error can't find the container with id 3fec637ea5ac5bf19bff2b647c39241959d924e29aa6cfe8b1b29f1ad1ca0864 Jul 15 10:46:24.094114 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.094091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" Jul 15 10:46:24.102624 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.102591 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01b503b_d02a_4ea2_8f1e_9fedf10e3d5a.slice/crio-a8a3f35768280ee2170c4853536b35f4c39989b896ae8898f58875631da01cfc WatchSource:0}: Error finding container a8a3f35768280ee2170c4853536b35f4c39989b896ae8898f58875631da01cfc: Status 404 returned error can't find the container with id a8a3f35768280ee2170c4853536b35f4c39989b896ae8898f58875631da01cfc Jul 15 10:46:24.116779 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.116743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gwd87" Jul 15 10:46:24.121622 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.121597 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-slrhd" Jul 15 10:46:24.126066 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.126036 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d807c3_969a_4305_a000_5eb13577cd18.slice/crio-7acf6eb124eceec41f6088d3ea19f91c58dfedf2799207c92a5054da2ad22201 WatchSource:0}: Error finding container 7acf6eb124eceec41f6088d3ea19f91c58dfedf2799207c92a5054da2ad22201: Status 404 returned error can't find the container with id 7acf6eb124eceec41f6088d3ea19f91c58dfedf2799207c92a5054da2ad22201 Jul 15 10:46:24.130093 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:46:24.130063 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26139f2_0422_42d7_b310_9cc1693f69ad.slice/crio-dc10bca3ebd6ae02f0c258917b27a3a159fc35afae364799b7212d64c7a85898 WatchSource:0}: Error finding container dc10bca3ebd6ae02f0c258917b27a3a159fc35afae364799b7212d64c7a85898: Status 404 returned error can't find the container with id dc10bca3ebd6ae02f0c258917b27a3a159fc35afae364799b7212d64c7a85898 Jul 15 10:46:24.209695 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.209601 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:46:24.370564 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.370523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:24.370764 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.370599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:24.370764 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.370715 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:24.370764 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.370760 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:24.371013 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.370834 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs podName:0119351c-bc27-4885-b458-ba76a4390657 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:25.370813465 +0000 UTC m=+3.147264690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs") pod "network-metrics-daemon-mczbg" (UID: "0119351c-bc27-4885-b458-ba76a4390657") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:24.371013 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.370857 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret podName:4f7beef7-c62f-468d-b02c-2a06d2322773 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:25.370846288 +0000 UTC m=+3.147297505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret") pod "global-pull-secret-syncer-d8q55" (UID: "4f7beef7-c62f-468d-b02c-2a06d2322773") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:24.471734 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.471576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:24.471926 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.471836 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:46:24.471926 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.471862 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:46:24.471926 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.471877 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf6ft for pod openshift-network-diagnostics/network-check-target-2fw2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:24.472152 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.471942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft podName:3439f5a7-5f74-422e-8089-c97186c19f74 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:25.471917844 +0000 UTC m=+3.248369077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf6ft" (UniqueName: "kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft") pod "network-check-target-2fw2b" (UID: "3439f5a7-5f74-422e-8089-c97186c19f74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:24.726085 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.725943 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:41:24 +0000 UTC" deadline="2028-04-16 06:55:48.354576712 +0000 UTC" Jul 15 10:46:24.726085 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.725981 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15380h9m23.628601969s" Jul 15 10:46:24.834513 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.832935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:24.834513 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:24.833130 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:24.849381 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.849283 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mqts8" event={"ID":"8c5d1637-944f-41b0-9af0-d93f9b31c9a1","Type":"ContainerStarted","Data":"a33de9d666e9f6af4d4e5486fc34b6db02520d61ea8f4440c3b90a70d0bbdb05"} Jul 15 10:46:24.859579 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.859511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9npvb" event={"ID":"a248225a-5a67-4ad2-9b38-fc3c04ee92ac","Type":"ContainerStarted","Data":"e8700c42192eb9a56a91b0c62804430e4fdb3892b7c31e51e5265aea0b613f15"} Jul 15 10:46:24.871039 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.870987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" event={"ID":"a6e6667c-5712-4706-943a-572aa76de3d3","Type":"ContainerStarted","Data":"ea0ca7d059b6ab91b82147ca0a5c9a4cdd58a2082192216f430883b67e693748"} Jul 15 10:46:24.880666 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.880619 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:46:24.884625 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.884582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" event={"ID":"6f592e6633d80c688b1a3245771d9573","Type":"ContainerStarted","Data":"216710675bab9c1e9cb8e83f88044afb17c2a5c505e86e3fb214da408bbfc35b"} Jul 15 10:46:24.889631 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.889577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-slrhd" event={"ID":"f26139f2-0422-42d7-b310-9cc1693f69ad","Type":"ContainerStarted","Data":"dc10bca3ebd6ae02f0c258917b27a3a159fc35afae364799b7212d64c7a85898"} Jul 15 10:46:24.895455 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.895249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s8dpc" event={"ID":"ec45a64a-a53a-4025-93dd-938828aa0269","Type":"ContainerStarted","Data":"3fec637ea5ac5bf19bff2b647c39241959d924e29aa6cfe8b1b29f1ad1ca0864"} Jul 15 10:46:24.918068 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.917966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j98bg" event={"ID":"41649f74-e757-4ee4-9df0-a63755eb1b0c","Type":"ContainerStarted","Data":"78322494c3719d88acd51f5cbee1141bf53464fae391815e720570d78aae112d"} Jul 15 10:46:24.922416 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.922375 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"18a55351bb135790ddb91e8f2e18f0214fdbb4bf5385db2f6a2f484ead134a66"} Jul 15 10:46:24.928227 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.928187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" event={"ID":"04c4f391b82cdea4e8f49294bf51b41d","Type":"ContainerStarted","Data":"fb9c3cdcfe1fcf64b2dc2691ff0f9b733c4f8664acb71fc63d61b3361d777adb"} Jul 15 10:46:24.949494 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.949382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerStarted","Data":"7acf6eb124eceec41f6088d3ea19f91c58dfedf2799207c92a5054da2ad22201"} Jul 15 10:46:24.963024 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:24.962983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" event={"ID":"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a","Type":"ContainerStarted","Data":"a8a3f35768280ee2170c4853536b35f4c39989b896ae8898f58875631da01cfc"} Jul 15 10:46:25.397623 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:25.397232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:25.397623 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:25.397306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:25.397623 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.397433 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:25.397623 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.397469 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:25.397623 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.397512 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret podName:4f7beef7-c62f-468d-b02c-2a06d2322773 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:27.397490097 +0000 UTC m=+5.173941320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret") pod "global-pull-secret-syncer-d8q55" (UID: "4f7beef7-c62f-468d-b02c-2a06d2322773") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:25.397623 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.397535 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs podName:0119351c-bc27-4885-b458-ba76a4390657 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:27.397523448 +0000 UTC m=+5.173974671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs") pod "network-metrics-daemon-mczbg" (UID: "0119351c-bc27-4885-b458-ba76a4390657") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:25.498187 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:25.498146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:25.498415 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.498349 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:46:25.498415 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.498382 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:46:25.498415 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.498396 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf6ft for pod openshift-network-diagnostics/network-check-target-2fw2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:25.498603 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.498458 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft podName:3439f5a7-5f74-422e-8089-c97186c19f74 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:27.498439463 +0000 UTC m=+5.274890694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf6ft" (UniqueName: "kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft") pod "network-check-target-2fw2b" (UID: "3439f5a7-5f74-422e-8089-c97186c19f74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:25.726808 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:25.726683 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:41:24 +0000 UTC" deadline="2027-12-12 01:35:29.56391153 +0000 UTC" Jul 15 10:46:25.726808 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:25.726752 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12350h49m3.83716425s" Jul 15 10:46:25.820449 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:25.820415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:25.820673 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.820564 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:25.820750 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:25.820663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:25.820813 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:25.820778 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:26.417674 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:26.417349 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 10:46:26.822818 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:26.822161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:26.822818 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:26.822315 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:27.414149 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:27.413505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:27.414149 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:27.413572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:27.414149 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.413723 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:27.414149 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.413743 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:27.414149 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.413792 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs podName:0119351c-bc27-4885-b458-ba76a4390657 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.413772132 +0000 UTC m=+9.190223365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs") pod "network-metrics-daemon-mczbg" (UID: "0119351c-bc27-4885-b458-ba76a4390657") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:27.414149 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.413814 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret podName:4f7beef7-c62f-468d-b02c-2a06d2322773 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.41380358 +0000 UTC m=+9.190254804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret") pod "global-pull-secret-syncer-d8q55" (UID: "4f7beef7-c62f-468d-b02c-2a06d2322773") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:27.515045 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:27.514361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:27.515045 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.514551 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:46:27.515045 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.514570 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:46:27.515045 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.514584 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf6ft for pod openshift-network-diagnostics/network-check-target-2fw2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:27.515045 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.514666 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft podName:3439f5a7-5f74-422e-8089-c97186c19f74 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.514646979 +0000 UTC m=+9.291098217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf6ft" (UniqueName: "kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft") pod "network-check-target-2fw2b" (UID: "3439f5a7-5f74-422e-8089-c97186c19f74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:27.820164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:27.820080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:27.820345 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.820225 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:27.820854 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:27.820642 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:27.820854 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:27.820748 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:28.820384 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:28.819782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:28.820384 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:28.819946 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:29.819874 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:29.819740 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:29.819874 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:29.819792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:29.821204 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:29.820339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:29.821653 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:29.821297 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:30.820970 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:30.820305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:30.820970 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:30.820468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:31.449918 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:31.449867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:31.450424 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:31.449953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:31.450424 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.450090 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:31.450424 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.450157 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret podName:4f7beef7-c62f-468d-b02c-2a06d2322773 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:39.450136493 +0000 UTC m=+17.226587710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret") pod "global-pull-secret-syncer-d8q55" (UID: "4f7beef7-c62f-468d-b02c-2a06d2322773") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:31.450612 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.450567 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:31.450697 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.450656 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs podName:0119351c-bc27-4885-b458-ba76a4390657 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:39.450635297 +0000 UTC m=+17.227086521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs") pod "network-metrics-daemon-mczbg" (UID: "0119351c-bc27-4885-b458-ba76a4390657") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:31.551044 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:31.551002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:31.551268 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.551192 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:46:31.551268 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.551217 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:46:31.551268 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.551231 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf6ft for pod openshift-network-diagnostics/network-check-target-2fw2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:31.551417 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.551292 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft podName:3439f5a7-5f74-422e-8089-c97186c19f74 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:39.551273438 +0000 UTC m=+17.327724666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf6ft" (UniqueName: "kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft") pod "network-check-target-2fw2b" (UID: "3439f5a7-5f74-422e-8089-c97186c19f74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:31.819781 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:31.819697 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:31.819951 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.819846 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:31.820085 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:31.820063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:31.820205 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:31.820167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:32.821098 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:32.821015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:32.821568 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:32.821164 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:33.820224 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:33.820152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:33.820415 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:33.820153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:33.820415 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:33.820277 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:33.820415 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:33.820365 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:34.822848 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:34.822816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:34.823302 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:34.822971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:35.819767 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:35.819735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:35.819927 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:35.819835 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:35.819927 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:35.819741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:35.820047 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:35.819997 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:36.819864 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:36.819773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:36.820295 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:36.819930 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:37.820061 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:37.820019 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:37.820538 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:37.820018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:37.820538 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:37.820159 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:37.820538 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:37.820258 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:38.820081 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:38.820042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:38.820490 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:38.820175 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:39.511777 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:39.511735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:39.511977 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:39.511794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:39.511977 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.511919 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:39.511977 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.511978 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs podName:0119351c-bc27-4885-b458-ba76a4390657 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:55.511963808 +0000 UTC m=+33.288415026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs") pod "network-metrics-daemon-mczbg" (UID: "0119351c-bc27-4885-b458-ba76a4390657") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:39.512145 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.511920 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:39.512145 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.512050 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret podName:4f7beef7-c62f-468d-b02c-2a06d2322773 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:55.512034087 +0000 UTC m=+33.288485317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret") pod "global-pull-secret-syncer-d8q55" (UID: "4f7beef7-c62f-468d-b02c-2a06d2322773") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:39.613098 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:39.613068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:39.613272 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.613208 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:46:39.613272 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.613226 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:46:39.613272 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.613236 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf6ft for pod openshift-network-diagnostics/network-check-target-2fw2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:39.613381 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.613288 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft podName:3439f5a7-5f74-422e-8089-c97186c19f74 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:55.613272705 +0000 UTC m=+33.389723922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf6ft" (UniqueName: "kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft") pod "network-check-target-2fw2b" (UID: "3439f5a7-5f74-422e-8089-c97186c19f74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:39.820358 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:39.820270 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:39.820856 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:39.820270 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:39.820856 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.820408 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:39.820856 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:39.820492 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:40.820006 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:40.819910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:40.820210 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:40.820039 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:41.819802 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:41.819767 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:41.820239 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:41.819888 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:41.820239 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:41.819944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:41.820239 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:41.820052 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:42.820271 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:42.820231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:42.820696 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:42.820360 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:43.819707 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:43.819664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:43.820031 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:43.819665 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:43.820031 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:43.819804 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:43.820031 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:43.819887 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:44.820463 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:44.820429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:44.820834 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:44.820548 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:45.819558 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:45.819362 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:45.819748 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:45.819364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:45.819748 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:45.819671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:45.819748 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:45.819714 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:46.016461 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.016426 2580 generic.go:356] "Generic (PLEG): container finished" podID="40d807c3-969a-4305-a000-5eb13577cd18" containerID="641be6c07654f92fc2cd650b1bd7e4f40bd3861ec31ef1800279ee7628685c87" exitCode=0 Jul 15 10:46:46.017299 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.016511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerDied","Data":"641be6c07654f92fc2cd650b1bd7e4f40bd3861ec31ef1800279ee7628685c87"} Jul 15 10:46:46.018243 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.018171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" event={"ID":"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a","Type":"ContainerStarted","Data":"598fd4846fb8462cd90e4eeefd4c78d71936810214fa4211cd4e714321523e4a"} Jul 15 10:46:46.019728 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.019661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9npvb" event={"ID":"a248225a-5a67-4ad2-9b38-fc3c04ee92ac","Type":"ContainerStarted","Data":"4832d9492db6157f2fa00394da740d8779f1e9f4b35f70a258f2748bdb00768f"} Jul 15 10:46:46.021510 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.021484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" event={"ID":"a6e6667c-5712-4706-943a-572aa76de3d3","Type":"ContainerStarted","Data":"545c9f714c22c7963ab0706c8d20b5fb632a19771a5f8e84ad0bdfc1fb21aa55"} Jul 15 10:46:46.025674 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.025643 2580 generic.go:356] "Generic (PLEG): container finished" podID="6f592e6633d80c688b1a3245771d9573" containerID="7d0daeae0d04e4d4fffe54ee1cc2ce6a24b9a65dbb5a652506b2a0fd018bc4e9" exitCode=0 Jul 15 10:46:46.025811 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.025714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" event={"ID":"6f592e6633d80c688b1a3245771d9573","Type":"ContainerDied","Data":"7d0daeae0d04e4d4fffe54ee1cc2ce6a24b9a65dbb5a652506b2a0fd018bc4e9"} Jul 15 10:46:46.027279 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.027253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-slrhd" event={"ID":"f26139f2-0422-42d7-b310-9cc1693f69ad","Type":"ContainerStarted","Data":"1bab967140acff60a7d4a009d3f05ef050d6aa4809cbc3f14f9718fc5efa0074"} Jul 15 10:46:46.028759 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.028730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s8dpc" event={"ID":"ec45a64a-a53a-4025-93dd-938828aa0269","Type":"ContainerStarted","Data":"1a6b4fde54b47121ef86118f4b9e3792a45122c4155f110f8fee46d8b8dfbd4f"} Jul 15 10:46:46.030199 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.030177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j98bg" event={"ID":"41649f74-e757-4ee4-9df0-a63755eb1b0c","Type":"ContainerStarted","Data":"4e3a2af8f308efa63a667d22a9c005a1fbaff3d29e86c3f7584212a936582223"} Jul 15 10:46:46.033311 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033295 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:46:46.033664 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033643 2580 generic.go:356] "Generic (PLEG): container finished" podID="f8874c1d-1f5f-4529-ac5b-bcec15627398" containerID="384e749ea862ea611f67bdcbab47b44f8afdf6043b5a96923a6ccedb7b8369ad" exitCode=1 Jul 15 10:46:46.033764 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"232db3f3a95ceb9abfba0eecfa18c726b2623fd52efc286e7f588a417aa437a1"} Jul 15 10:46:46.033828 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033769 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"1f1557d773c4954f5b40c24faff757c82055c18dddfae86ddcea52e4b8b2910b"} Jul 15 10:46:46.033882 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"85f80a0a645b0fee8f2bd13fe6a8757585a79be1757a0a7e36cceeb55d7f48a0"} Jul 15 10:46:46.033882 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"dedead0336bac7d13ff3778d07645e723b34224bc268c1f4c2db8251e86762f2"} Jul 15 10:46:46.033983 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033922 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerDied","Data":"384e749ea862ea611f67bdcbab47b44f8afdf6043b5a96923a6ccedb7b8369ad"} Jul 15 10:46:46.033983 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.033974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"79ba70cf25a4d99d47ce170d61b245f5f6c5a32deef3f547ea9a26729ea0c861"} Jul 15 10:46:46.035124 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.035103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" event={"ID":"04c4f391b82cdea4e8f49294bf51b41d","Type":"ContainerStarted","Data":"6eed78ae3a886c75eb58778c5e30f5cf6940ff81afde36b5e660f6aff2124573"} Jul 15 10:46:46.062519 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.062336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:46.062912 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.062896 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:46.069758 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.069708 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s8dpc" podStartSLOduration=2.146913048 podStartE2EDuration="23.069692955s" totalImagesPullingTime="20.922779907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:46.069545038 +0000 UTC m=+23.845996276" watchObservedRunningTime="2026-07-15 10:46:46.069692955 +0000 UTC m=+23.846144192" Jul 15 10:46:46.092275 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.092216 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-slrhd" podStartSLOduration=2.21573237 podStartE2EDuration="23.092168043s" totalImagesPullingTime="20.876435673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:46.091335241 +0000 UTC m=+23.867786481" watchObservedRunningTime="2026-07-15 10:46:46.092168043 +0000 UTC m=+23.868619280" Jul 15 10:46:46.093599 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.093552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:46.131039 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.130990 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-140.ec2.internal" podStartSLOduration=22.130972659 podStartE2EDuration="22.130972659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:46.112725304 +0000 UTC m=+23.889176540" watchObservedRunningTime="2026-07-15 10:46:46.130972659 +0000 UTC m=+23.907423896" Jul 15 10:46:46.178233 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.178186 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4tvsj" podStartSLOduration=2.411932315 podStartE2EDuration="23.178169521s" totalImagesPullingTime="20.766237206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:46.152706337 +0000 UTC m=+23.929157585" watchObservedRunningTime="2026-07-15 10:46:46.178169521 +0000 UTC m=+23.954620757" Jul 15 10:46:46.178428 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.178293 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9npvb" podStartSLOduration=2.441090558 podStartE2EDuration="23.178289987s" totalImagesPullingTime="20.737199429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:46.177933981 +0000 UTC m=+23.954385217" watchObservedRunningTime="2026-07-15 10:46:46.178289987 +0000 UTC m=+23.954741221" Jul 15 10:46:46.194416 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.194360 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j98bg" podStartSLOduration=2.258282815 podStartE2EDuration="23.194345626s" totalImagesPullingTime="20.936062811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:46.194139427 +0000 UTC m=+23.970590675" watchObservedRunningTime="2026-07-15 10:46:46.194345626 +0000 UTC m=+23.970796862" Jul 15 10:46:46.330522 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.330423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:46.428725 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.428690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:46.671796 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.671744 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:46.774056 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.774023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:46.820493 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:46.820450 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:46.820680 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:46.820620 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:47.039602 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.039495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" event={"ID":"6f592e6633d80c688b1a3245771d9573","Type":"ContainerStarted","Data":"af81fac05dc47350562d010b9dffe87273f403e79d8cf2b569b32a5b92e70333"} Jul 15 10:46:47.041197 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.041153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mqts8" event={"ID":"8c5d1637-944f-41b0-9af0-d93f9b31c9a1","Type":"ContainerStarted","Data":"0a68d430f1a5bcb70fe20b2af21d09e90a0067f6eef477bf532beceed31a8e12"} Jul 15 10:46:47.042114 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.042086 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:47.042620 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.042603 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s8dpc" Jul 15 10:46:47.089455 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.089389 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-140.ec2.internal" podStartSLOduration=23.089370111 podStartE2EDuration="23.089370111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:47.062627879 +0000 UTC m=+24.839079135" watchObservedRunningTime="2026-07-15 10:46:47.089370111 +0000 UTC m=+24.865821359" Jul 15 10:46:47.090302 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.090259 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mqts8" podStartSLOduration=3.156346273 podStartE2EDuration="24.090248254s" totalImagesPullingTime="20.933901981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:47.089852498 +0000 UTC m=+24.866303736" watchObservedRunningTime="2026-07-15 10:46:47.090248254 +0000 UTC m=+24.866699493" Jul 15 10:46:47.676646 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.676559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:47.729426 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.729401 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 10:46:47.732259 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.732153 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T10:46:47.729422027Z","UUID":"de2dbf00-a9dc-4a71-ab98-37b420107cdb","Handler":null,"Name":"","Endpoint":""} Jul 15 10:46:47.733864 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.733841 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 10:46:47.733984 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.733890 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 10:46:47.772909 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.772830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:47.820169 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.820133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:47.820349 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:47.820267 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:47.820349 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.820322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:47.820458 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:47.820436 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:47.919477 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:47.919362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:48.045647 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:48.045608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" event={"ID":"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a","Type":"ContainerStarted","Data":"21d47715acfc16c1a613458889708c1667dab7db450c079df600fa483e03c55a"} Jul 15 10:46:48.560287 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:48.560194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:48.616698 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:48.616663 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:48.820030 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:48.819950 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:48.820185 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:48.820109 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:48.969237 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:48.969200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:49.051752 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:49.051718 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:49.052166 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:49.051914 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:46:49.052449 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:49.052423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"b27337aeae4b502ab45a79a9b015f9b7c364f271061fea4f83c64d099300f780"} Jul 15 10:46:49.109876 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:49.109841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:49.819553 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:49.819508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:49.819763 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:49.819507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:49.819763 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:49.819653 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:49.819872 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:49.819796 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:50.057256 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:50.057227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:50.819965 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:50.819669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:50.820098 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:50.820033 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:51.058740 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.058705 2580 generic.go:356] "Generic (PLEG): container finished" podID="40d807c3-969a-4305-a000-5eb13577cd18" containerID="c542d6955639e6527a1ed2cbb04c65b05f37f2e8a17c9dad7949ccc2ca5364f7" exitCode=0 Jul 15 10:46:51.059383 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.058792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerDied","Data":"c542d6955639e6527a1ed2cbb04c65b05f37f2e8a17c9dad7949ccc2ca5364f7"} Jul 15 10:46:51.061004 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.060978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" event={"ID":"b01b503b-d02a-4ea2-8f1e-9fedf10e3d5a","Type":"ContainerStarted","Data":"956f1b98b2de5f9bb50fb507e21eb1bafb22192d6332a44c5d931c007fe02481"} Jul 15 10:46:51.064433 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.064415 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:46:51.064743 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.064725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"61a79f10ef702e001f8afa902f9dbc62bdebd602ff545eb5f222297589f63969"} Jul 15 10:46:51.065034 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.065002 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:51.065122 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.065069 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:51.065177 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.065168 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:51.065226 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.065183 2580 scope.go:117] "RemoveContainer" containerID="384e749ea862ea611f67bdcbab47b44f8afdf6043b5a96923a6ccedb7b8369ad" Jul 15 10:46:51.081413 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.081389 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:51.081554 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.081488 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:46:51.110005 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.109944 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bm6fk" podStartSLOduration=2.720080776 podStartE2EDuration="28.109925757s" totalImagesPullingTime="25.389844981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:51.108786747 +0000 UTC m=+28.885238005" watchObservedRunningTime="2026-07-15 10:46:51.109925757 +0000 UTC m=+28.886376972" Jul 15 10:46:51.250248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.250212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:51.819734 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.819660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:51.820040 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.819664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:51.820040 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:51.819768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:51.820040 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:51.819846 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:51.881841 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.881804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:51.954549 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:51.954516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:52.009976 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.009941 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:52.070680 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.070596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:52.071472 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.071456 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:46:52.071882 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.071856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" event={"ID":"f8874c1d-1f5f-4529-ac5b-bcec15627398","Type":"ContainerStarted","Data":"7ca9ef8e46c7b985d03a53d0f31b7a18942bee6f2384f8972f3a6fa8f9bd825b"} Jul 15 10:46:52.106675 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.106620 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" podStartSLOduration=7.809531848 podStartE2EDuration="29.106603377s" totalImagesPullingTime="21.297071529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:52.105439069 +0000 UTC m=+29.881890308" watchObservedRunningTime="2026-07-15 10:46:52.106603377 +0000 UTC m=+29.883054610" Jul 15 10:46:52.155315 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.155282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:52.345877 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.345691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:52.436256 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.436220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:52.505721 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.505687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mczbg"] Jul 15 10:46:52.505956 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.505820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:52.505956 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:52.505920 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:52.508731 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.508698 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d8q55"] Jul 15 10:46:52.508882 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.508809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:52.508949 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:52.508917 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:52.509515 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.509483 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2fw2b"] Jul 15 10:46:52.509656 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.509568 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:52.509723 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:52.509682 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:52.589513 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:52.589480 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:53.075961 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.075871 2580 generic.go:356] "Generic (PLEG): container finished" podID="40d807c3-969a-4305-a000-5eb13577cd18" containerID="29e6ec8437b47d3def8f4ce5c01a50b4ad5cc389d438461c540f24cb4229f718" exitCode=0 Jul 15 10:46:53.076361 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.075963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerDied","Data":"29e6ec8437b47d3def8f4ce5c01a50b4ad5cc389d438461c540f24cb4229f718"} Jul 15 10:46:53.538342 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.538301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:53.637838 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.637792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:53.793487 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.793403 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:53.819673 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.819644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:53.819673 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.819687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:53.819919 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:53.819776 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:53.819919 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:53.819850 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:53.986122 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:53.986089 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:54.391750 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:54.391709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:54.597706 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:54.597676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:54.819653 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:54.819612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:54.819844 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:54.819752 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:55.022648 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.022545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:55.074781 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.074747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:55.083691 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.083662 2580 generic.go:356] "Generic (PLEG): container finished" podID="40d807c3-969a-4305-a000-5eb13577cd18" containerID="11db8a9fdd0bdcb06f2f015c31460b870c92c06b83299486b1e4d2291f25fe14" exitCode=0 Jul 15 10:46:55.083863 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.083724 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerDied","Data":"11db8a9fdd0bdcb06f2f015c31460b870c92c06b83299486b1e4d2291f25fe14"} Jul 15 10:46:55.187862 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.187831 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:55.401718 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.401683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:55.529008 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.528970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:55.529212 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.529075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:55.529212 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.529144 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:55.529212 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.529169 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:55.529375 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.529218 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret podName:4f7beef7-c62f-468d-b02c-2a06d2322773 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:27.529202595 +0000 UTC m=+65.305653812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret") pod "global-pull-secret-syncer-d8q55" (UID: "4f7beef7-c62f-468d-b02c-2a06d2322773") : object "kube-system"/"original-pull-secret" not registered Jul 15 10:46:55.529375 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.529232 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs podName:0119351c-bc27-4885-b458-ba76a4390657 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:27.529225629 +0000 UTC m=+65.305676844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs") pod "network-metrics-daemon-mczbg" (UID: "0119351c-bc27-4885-b458-ba76a4390657") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 10:46:55.590110 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.590077 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:55.630272 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.630240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:55.630435 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.630418 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 10:46:55.630501 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.630443 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 10:46:55.630501 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.630459 2580 projected.go:196] Error preparing data for projected volume kube-api-access-nf6ft for pod openshift-network-diagnostics/network-check-target-2fw2b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:55.630619 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.630525 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft podName:3439f5a7-5f74-422e-8089-c97186c19f74 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:27.630506737 +0000 UTC m=+65.406957953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nf6ft" (UniqueName: "kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft") pod "network-check-target-2fw2b" (UID: "3439f5a7-5f74-422e-8089-c97186c19f74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 10:46:55.788863 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.788778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:55.819787 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.819753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:55.819985 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.819804 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:55.819985 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.819898 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:55.820100 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:55.820003 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:55.889474 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.889428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:55.989278 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:55.989238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:56.594752 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:56.594715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:56.696098 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:56.696063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:56.819676 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:56.819645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:56.819831 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:56.819782 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:57.193804 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:57.193759 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:57.389720 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:57.389687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:57.492182 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:57.492092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:57.591750 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:57.591718 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:57.820018 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:57.819927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:57.820018 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:57.819956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:57.820452 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:57.820064 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:57.820452 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:57.820176 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:58.326946 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:58.326912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:58.427061 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:58.427012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:58.494230 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:58.494199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:58.819470 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:58.819420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:46:58.819684 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:58.819553 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:46:59.108896 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:59.108863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:59.389049 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:59.388966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:59.601126 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:59.601097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:59.793201 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:59.793170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:46:59.819357 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:59.819322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:46:59.819357 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:59.819357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:46:59.819553 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:59.819455 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:46:59.819623 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:46:59.819591 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:46:59.989046 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:46:59.988960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:00.097177 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:00.097139 2580 generic.go:356] "Generic (PLEG): container finished" podID="40d807c3-969a-4305-a000-5eb13577cd18" containerID="ac2355eb730346876eb8cd8472ed2cc975f799d440c2e62c68b607e224f83eca" exitCode=0 Jul 15 10:47:00.097329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:00.097186 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerDied","Data":"ac2355eb730346876eb8cd8472ed2cc975f799d440c2e62c68b607e224f83eca"} Jul 15 10:47:00.192661 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:00.192394 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:00.820055 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:00.820019 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:00.820271 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:00.820135 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:47:01.102783 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:01.102749 2580 generic.go:356] "Generic (PLEG): container finished" podID="40d807c3-969a-4305-a000-5eb13577cd18" containerID="be8d56ebe1c5370ba2737c5079a76a19ada44ff4ce451caec7ac26da2ee68e8d" exitCode=0 Jul 15 10:47:01.102947 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:01.102802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerDied","Data":"be8d56ebe1c5370ba2737c5079a76a19ada44ff4ce451caec7ac26da2ee68e8d"} Jul 15 10:47:01.159776 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:01.159702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:01.820063 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:01.820022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:01.820526 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:01.820138 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:47:01.820526 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:01.820200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:01.820526 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:01.820320 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:47:01.946704 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:01.946668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:02.048765 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.048712 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:02.107759 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.107726 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwd87" event={"ID":"40d807c3-969a-4305-a000-5eb13577cd18","Type":"ContainerStarted","Data":"9b6a5cb1bdd02d9e6b7f8cbd770844e01a51e67d5e8990c789b690a6222e10a6"} Jul 15 10:47:02.132044 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.131990 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gwd87" podStartSLOduration=7.217968591 podStartE2EDuration="39.131971616s" totalImagesPullingTime="31.914003025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:02.131759509 +0000 UTC m=+39.908210748" watchObservedRunningTime="2026-07-15 10:47:02.131971616 +0000 UTC m=+39.908422851" Jul 15 10:47:02.395481 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.395375 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:02.592136 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.592101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:02.787179 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.787083 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:02.820857 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.820828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:02.821269 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:02.820920 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:47:02.892545 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.892503 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:02.951314 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:02.951286 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:03.047341 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:03.047261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:03.295751 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:03.293671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:03.352032 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:03.351992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:03.662403 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:03.662317 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:03.820025 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:03.819973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:03.820025 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:03.820015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:03.820234 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:03.820115 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:47:03.820234 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:03.820220 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:47:04.820027 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:04.819987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:04.820410 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:04.820102 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:47:05.820292 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:05.820256 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:05.820727 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:05.820256 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:05.820727 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:05.820368 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:47:05.820727 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:05.820448 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:47:06.820141 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:06.820103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:06.820395 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:06.820219 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:47:07.820013 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:07.819975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:07.820217 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:07.819976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:07.820217 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:07.820086 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:47:07.820217 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:07.820199 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:47:08.819660 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:08.819626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:08.820106 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:08.819740 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8q55" podUID="4f7beef7-c62f-468d-b02c-2a06d2322773" Jul 15 10:47:08.874342 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:08.874303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:08.963310 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:08.963270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:09.060771 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.060738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:09.161773 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.161725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:09.669701 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.669672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:09.734924 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.734889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:09.820327 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.820294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:09.820730 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.820294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:09.820730 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:09.820415 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mczbg" podUID="0119351c-bc27-4885-b458-ba76a4390657" Jul 15 10:47:09.820730 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:09.820469 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2fw2b" podUID="3439f5a7-5f74-422e-8089-c97186c19f74" Jul 15 10:47:09.872507 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.872480 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-140.ec2.internal" event="NodeReady" Jul 15 10:47:09.872700 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.872619 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 10:47:09.909034 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.909003 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84c8f5d4dd-g7hxg"] Jul 15 10:47:09.930804 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.930727 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mz6kd"] Jul 15 10:47:09.930932 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.930866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.933585 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.933547 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 10:47:09.933585 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.933568 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 10:47:09.933804 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.933591 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 10:47:09.933804 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.933638 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4jkx4\"" Jul 15 10:47:09.938108 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.938084 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.938921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f48df869-59d0-472d-9b6a-a4df786659ae-ca-trust-extracted\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.938962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f48df869-59d0-472d-9b6a-a4df786659ae-image-registry-private-configuration\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.938990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f48df869-59d0-472d-9b6a-a4df786659ae-installation-pull-secrets\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.939016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-bound-sa-token\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.939074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f48df869-59d0-472d-9b6a-a4df786659ae-registry-certificates\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.939104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f48df869-59d0-472d-9b6a-a4df786659ae-trusted-ca\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.939162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-registry-tls\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.939248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.939186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6txn\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-kube-api-access-b6txn\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:09.946132 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.946105 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84c8f5d4dd-g7hxg"] Jul 15 10:47:09.946132 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.946138 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ws2z8"] Jul 15 10:47:09.946344 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.946204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:09.948585 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.948548 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 10:47:09.948778 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.948757 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 10:47:09.948778 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.948775 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sw5h4\"" Jul 15 10:47:09.968281 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.968246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mz6kd"] Jul 15 10:47:09.968281 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.968287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ws2z8"] Jul 15 10:47:09.968521 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.968385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:09.970797 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.970772 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 10:47:09.970941 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.970782 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 10:47:09.971014 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.970997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 10:47:09.971057 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.971034 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 10:47:09.971253 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:09.971241 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ghxdw\"" Jul 15 10:47:10.028635 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.028599 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tjz76"] Jul 15 10:47:10.029754 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.029726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:10.039939 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.039902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-registry-tls\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.039939 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.039941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6txn\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-kube-api-access-b6txn\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.040139 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.039965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hps7\" (UniqueName: \"kubernetes.io/projected/ae568f44-9c3f-46fa-a508-22bea4aa54ae-kube-api-access-4hps7\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.040139 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.039989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f48df869-59d0-472d-9b6a-a4df786659ae-ca-trust-extracted\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.040139 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae568f44-9c3f-46fa-a508-22bea4aa54ae-crio-socket\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.040139 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f48df869-59d0-472d-9b6a-a4df786659ae-image-registry-private-configuration\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.040139 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f48df869-59d0-472d-9b6a-a4df786659ae-installation-pull-secrets\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.040417 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-bound-sa-token\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.040619 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f48df869-59d0-472d-9b6a-a4df786659ae-ca-trust-extracted\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.040805 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0668642b-02ed-4dbe-8081-abfc923974a5-metrics-tls\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.040805 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae568f44-9c3f-46fa-a508-22bea4aa54ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.040805 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0668642b-02ed-4dbe-8081-abfc923974a5-config-volume\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.040805 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0668642b-02ed-4dbe-8081-abfc923974a5-tmp-dir\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.040805 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae568f44-9c3f-46fa-a508-22bea4aa54ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.041029 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f48df869-59d0-472d-9b6a-a4df786659ae-registry-certificates\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.041029 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae568f44-9c3f-46fa-a508-22bea4aa54ae-data-volume\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.041029 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f48df869-59d0-472d-9b6a-a4df786659ae-trusted-ca\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.041029 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.040930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7472b\" (UniqueName: \"kubernetes.io/projected/0668642b-02ed-4dbe-8081-abfc923974a5-kube-api-access-7472b\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.041788 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.041764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f48df869-59d0-472d-9b6a-a4df786659ae-registry-certificates\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.042234 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.042209 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f48df869-59d0-472d-9b6a-a4df786659ae-trusted-ca\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.044192 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.044169 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tjz76"] Jul 15 10:47:10.044318 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.044265 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.045224 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.045200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f48df869-59d0-472d-9b6a-a4df786659ae-image-registry-private-configuration\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.045318 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.045200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f48df869-59d0-472d-9b6a-a4df786659ae-installation-pull-secrets\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.045381 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.045348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-registry-tls\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.047488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.047463 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 10:47:10.047617 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.047503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ngctx\"" Jul 15 10:47:10.047617 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.047506 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 10:47:10.047617 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.047509 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 10:47:10.054883 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.054855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6txn\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-kube-api-access-b6txn\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.056696 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.056676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f48df869-59d0-472d-9b6a-a4df786659ae-bound-sa-token\") pod \"image-registry-84c8f5d4dd-g7hxg\" (UID: \"f48df869-59d0-472d-9b6a-a4df786659ae\") " pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.118182 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.118142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:10.142016 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.141981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hps7\" (UniqueName: \"kubernetes.io/projected/ae568f44-9c3f-46fa-a508-22bea4aa54ae-kube-api-access-4hps7\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.142016 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae568f44-9c3f-46fa-a508-22bea4aa54ae-crio-socket\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0668642b-02ed-4dbe-8081-abfc923974a5-metrics-tls\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae568f44-9c3f-46fa-a508-22bea4aa54ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0668642b-02ed-4dbe-8081-abfc923974a5-config-volume\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0668642b-02ed-4dbe-8081-abfc923974a5-tmp-dir\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae568f44-9c3f-46fa-a508-22bea4aa54ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d556996-8ed1-44c3-b41a-adf835e56a26-cert\") pod \"ingress-canary-tjz76\" (UID: \"2d556996-8ed1-44c3-b41a-adf835e56a26\") " pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7kq\" (UniqueName: \"kubernetes.io/projected/2d556996-8ed1-44c3-b41a-adf835e56a26-kube-api-access-tl7kq\") pod \"ingress-canary-tjz76\" (UID: \"2d556996-8ed1-44c3-b41a-adf835e56a26\") " pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae568f44-9c3f-46fa-a508-22bea4aa54ae-crio-socket\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.142249 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae568f44-9c3f-46fa-a508-22bea4aa54ae-data-volume\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.142708 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7472b\" (UniqueName: \"kubernetes.io/projected/0668642b-02ed-4dbe-8081-abfc923974a5-kube-api-access-7472b\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.142708 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae568f44-9c3f-46fa-a508-22bea4aa54ae-data-volume\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.142708 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0668642b-02ed-4dbe-8081-abfc923974a5-tmp-dir\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.142859 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0668642b-02ed-4dbe-8081-abfc923974a5-config-volume\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.142859 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.142812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae568f44-9c3f-46fa-a508-22bea4aa54ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.144601 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.144557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0668642b-02ed-4dbe-8081-abfc923974a5-metrics-tls\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.144673 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.144647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae568f44-9c3f-46fa-a508-22bea4aa54ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.150616 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.150585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7472b\" (UniqueName: \"kubernetes.io/projected/0668642b-02ed-4dbe-8081-abfc923974a5-kube-api-access-7472b\") pod \"dns-default-mz6kd\" (UID: \"0668642b-02ed-4dbe-8081-abfc923974a5\") " pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.150886 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.150866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hps7\" (UniqueName: \"kubernetes.io/projected/ae568f44-9c3f-46fa-a508-22bea4aa54ae-kube-api-access-4hps7\") pod \"insights-runtime-extractor-ws2z8\" (UID: \"ae568f44-9c3f-46fa-a508-22bea4aa54ae\") " pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.195180 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.195093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:10.242275 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.242241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:10.243457 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.243424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d556996-8ed1-44c3-b41a-adf835e56a26-cert\") pod \"ingress-canary-tjz76\" (UID: \"2d556996-8ed1-44c3-b41a-adf835e56a26\") " pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.243593 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.243464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7kq\" (UniqueName: \"kubernetes.io/projected/2d556996-8ed1-44c3-b41a-adf835e56a26-kube-api-access-tl7kq\") pod \"ingress-canary-tjz76\" (UID: \"2d556996-8ed1-44c3-b41a-adf835e56a26\") " pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.245846 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.245823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d556996-8ed1-44c3-b41a-adf835e56a26-cert\") pod \"ingress-canary-tjz76\" (UID: \"2d556996-8ed1-44c3-b41a-adf835e56a26\") " pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.248756 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.248731 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:10.251387 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.251361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7kq\" (UniqueName: \"kubernetes.io/projected/2d556996-8ed1-44c3-b41a-adf835e56a26-kube-api-access-tl7kq\") pod \"ingress-canary-tjz76\" (UID: \"2d556996-8ed1-44c3-b41a-adf835e56a26\") " pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.256110 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.256094 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:10.277163 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.277142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ws2z8" Jul 15 10:47:10.369478 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.369006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tjz76" Jul 15 10:47:10.494422 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.494387 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84c8f5d4dd-g7hxg"] Jul 15 10:47:10.498651 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:10.498624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48df869_59d0_472d_9b6a_a4df786659ae.slice/crio-20990a9ec6884efccc4ede5835ad944304643bb3f2cfdb3cad1dc72cbfbceaca WatchSource:0}: Error finding container 20990a9ec6884efccc4ede5835ad944304643bb3f2cfdb3cad1dc72cbfbceaca: Status 404 returned error can't find the container with id 20990a9ec6884efccc4ede5835ad944304643bb3f2cfdb3cad1dc72cbfbceaca Jul 15 10:47:10.506148 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.506123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mz6kd"] Jul 15 10:47:10.509551 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:10.509520 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0668642b_02ed_4dbe_8081_abfc923974a5.slice/crio-15dd5eb94c72610376c338b6762fecb22985ca5c435c3f951a007508181f635b WatchSource:0}: Error finding container 15dd5eb94c72610376c338b6762fecb22985ca5c435c3f951a007508181f635b: Status 404 returned error can't find the container with id 15dd5eb94c72610376c338b6762fecb22985ca5c435c3f951a007508181f635b Jul 15 10:47:10.511282 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.511256 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ws2z8"] Jul 15 10:47:10.514915 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:10.514891 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae568f44_9c3f_46fa_a508_22bea4aa54ae.slice/crio-39ba148c58945e1b4a9bdf36e27dcaaeb2d048db01acf8dfd3600d29bba05b94 WatchSource:0}: Error finding container 39ba148c58945e1b4a9bdf36e27dcaaeb2d048db01acf8dfd3600d29bba05b94: Status 404 returned error can't find the container with id 39ba148c58945e1b4a9bdf36e27dcaaeb2d048db01acf8dfd3600d29bba05b94 Jul 15 10:47:10.555742 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.555714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tjz76"] Jul 15 10:47:10.560457 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:10.560423 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d556996_8ed1_44c3_b41a_adf835e56a26.slice/crio-8df3a735998c31c5993646a88050afa6c591090fc15063384c361a512e4e992d WatchSource:0}: Error finding container 8df3a735998c31c5993646a88050afa6c591090fc15063384c361a512e4e992d: Status 404 returned error can't find the container with id 8df3a735998c31c5993646a88050afa6c591090fc15063384c361a512e4e992d Jul 15 10:47:10.820065 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.819948 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:10.822613 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:10.822590 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 10:47:11.128911 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.128872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ws2z8" event={"ID":"ae568f44-9c3f-46fa-a508-22bea4aa54ae","Type":"ContainerStarted","Data":"39ba148c58945e1b4a9bdf36e27dcaaeb2d048db01acf8dfd3600d29bba05b94"} Jul 15 10:47:11.129907 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.129882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mz6kd" event={"ID":"0668642b-02ed-4dbe-8081-abfc923974a5","Type":"ContainerStarted","Data":"15dd5eb94c72610376c338b6762fecb22985ca5c435c3f951a007508181f635b"} Jul 15 10:47:11.131188 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.131165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" event={"ID":"f48df869-59d0-472d-9b6a-a4df786659ae","Type":"ContainerStarted","Data":"c1bbf3648de793c447813cc05c8ea4e1c87844c2bb9623f62b184996f59807c6"} Jul 15 10:47:11.131290 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.131206 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" event={"ID":"f48df869-59d0-472d-9b6a-a4df786659ae","Type":"ContainerStarted","Data":"20990a9ec6884efccc4ede5835ad944304643bb3f2cfdb3cad1dc72cbfbceaca"} Jul 15 10:47:11.131290 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.131264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:11.132168 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.132142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tjz76" event={"ID":"2d556996-8ed1-44c3-b41a-adf835e56a26","Type":"ContainerStarted","Data":"8df3a735998c31c5993646a88050afa6c591090fc15063384c361a512e4e992d"} Jul 15 10:47:11.155284 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.155238 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" podStartSLOduration=3.155222485 podStartE2EDuration="3.155222485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:11.154215734 +0000 UTC m=+48.930667001" watchObservedRunningTime="2026-07-15 10:47:11.155222485 +0000 UTC m=+48.931673722" Jul 15 10:47:11.820254 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.820212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:11.821241 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.820713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:11.826918 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.826880 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 10:47:11.827328 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.826984 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qpwxt\"" Jul 15 10:47:11.827537 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.827481 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlzgw\"" Jul 15 10:47:11.831285 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.827676 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 10:47:11.831894 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:11.831487 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 10:47:12.272904 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:12.272832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:12.329333 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:12.329293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:18.154341 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:18.154307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ws2z8" event={"ID":"ae568f44-9c3f-46fa-a508-22bea4aa54ae","Type":"ContainerStarted","Data":"671a860cd3257dff9f25e0ccd83c89a2263e4398bf029d211589b4f1d01ee735"} Jul 15 10:47:18.156559 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:18.156524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mz6kd" event={"ID":"0668642b-02ed-4dbe-8081-abfc923974a5","Type":"ContainerStarted","Data":"133a12842e65da4e18d745d280cd7533dfca216f257bc796e91424ad214a3c9a"} Jul 15 10:47:18.156763 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:18.156626 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:18.156763 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:18.156667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mz6kd" event={"ID":"0668642b-02ed-4dbe-8081-abfc923974a5","Type":"ContainerStarted","Data":"0a93010571aeaefd99198d46ed918da8d6bbc7bdd1a7f4e585536e64a854191b"} Jul 15 10:47:18.158210 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:18.158183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tjz76" event={"ID":"2d556996-8ed1-44c3-b41a-adf835e56a26","Type":"ContainerStarted","Data":"dffdb39375fcb707d23304ec755a89b15a6e4b3fc577f5d0e54b515270dc6101"} Jul 15 10:47:18.183207 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:18.183147 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mz6kd" podStartSLOduration=2.017558786 podStartE2EDuration="9.18312654s" totalImagesPullingTime="7.165567754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:18.182453911 +0000 UTC m=+55.958905172" watchObservedRunningTime="2026-07-15 10:47:18.18312654 +0000 UTC m=+55.959577777" Jul 15 10:47:18.204730 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:18.204673 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tjz76" podStartSLOduration=1.088993611 podStartE2EDuration="8.204658233s" totalImagesPullingTime="7.115664622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:18.203271101 +0000 UTC m=+55.979722409" watchObservedRunningTime="2026-07-15 10:47:18.204658233 +0000 UTC m=+55.981109470" Jul 15 10:47:19.277290 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:19.277250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:19.371314 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:19.371278 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:19.492992 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:19.492951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:19.813585 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:19.813547 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:19.893793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:19.893756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:20.010023 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:20.009992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:20.110926 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:20.110888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:23.089066 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:23.089029 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22ll8" Jul 15 10:47:23.177923 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:23.177885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ws2z8" event={"ID":"ae568f44-9c3f-46fa-a508-22bea4aa54ae","Type":"ContainerStarted","Data":"4bdf460ade570ff788cdb1e226c32f00aea655656d4e62e3771f998d1f7719ad"} Jul 15 10:47:23.273019 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:23.272982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:23.367227 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:23.367117 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:23.435810 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:23.435764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:23.886997 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:23.886960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:24.101672 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:24.101637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:24.203557 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:24.203442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:24.514519 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:24.514441 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:24.608281 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:24.608238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:24.669535 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:24.669502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:25.023068 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:25.022954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:25.123542 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:25.123507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:25.679133 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:25.679094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:25.780704 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:25.780676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:26.191929 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:26.191893 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ws2z8" event={"ID":"ae568f44-9c3f-46fa-a508-22bea4aa54ae","Type":"ContainerStarted","Data":"81a2430121dfc43c8c378c683e05821628bbafa495deeebe0f429e8ebd9594bd"} Jul 15 10:47:26.220138 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:26.220084 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ws2z8" podStartSLOduration=2.219848165 podStartE2EDuration="17.220064429s" totalImagesPullingTime="15.000216264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:47:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:26.211709547 +0000 UTC m=+63.988160784" watchObservedRunningTime="2026-07-15 10:47:26.220064429 +0000 UTC m=+63.996515645" Jul 15 10:47:27.585520 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.585477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:27.585919 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.585528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:27.588220 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.588195 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 10:47:27.589008 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.588993 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 10:47:27.598070 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.598038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119351c-bc27-4885-b458-ba76a4390657-metrics-certs\") pod \"network-metrics-daemon-mczbg\" (UID: \"0119351c-bc27-4885-b458-ba76a4390657\") " pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:27.598207 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.598040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f7beef7-c62f-468d-b02c-2a06d2322773-original-pull-secret\") pod \"global-pull-secret-syncer-d8q55\" (UID: \"4f7beef7-c62f-468d-b02c-2a06d2322773\") " pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:27.629206 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.629169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8q55" Jul 15 10:47:27.686932 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.686899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:27.689754 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.689728 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 10:47:27.699819 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.699794 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 10:47:27.712526 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.712476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6ft\" (UniqueName: \"kubernetes.io/projected/3439f5a7-5f74-422e-8089-c97186c19f74-kube-api-access-nf6ft\") pod \"network-check-target-2fw2b\" (UID: \"3439f5a7-5f74-422e-8089-c97186c19f74\") " pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:27.742388 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.742360 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qpwxt\"" Jul 15 10:47:27.750392 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.750360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mczbg" Jul 15 10:47:27.750696 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.750678 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlzgw\"" Jul 15 10:47:27.756703 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.756631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d8q55"] Jul 15 10:47:27.759599 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.759562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:27.760131 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:27.760103 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7beef7_c62f_468d_b02c_2a06d2322773.slice/crio-4279b095b89db4913c8b94bef28edecd793d4a38e68582af9b304ce257b36337 WatchSource:0}: Error finding container 4279b095b89db4913c8b94bef28edecd793d4a38e68582af9b304ce257b36337: Status 404 returned error can't find the container with id 4279b095b89db4913c8b94bef28edecd793d4a38e68582af9b304ce257b36337 Jul 15 10:47:27.887072 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.887039 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mczbg"] Jul 15 10:47:27.889810 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:27.889787 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0119351c_bc27_4885_b458_ba76a4390657.slice/crio-155640c10ae90766796c1953d42d17a9a34ead0f37be99bb83808d2b97a5c185 WatchSource:0}: Error finding container 155640c10ae90766796c1953d42d17a9a34ead0f37be99bb83808d2b97a5c185: Status 404 returned error can't find the container with id 155640c10ae90766796c1953d42d17a9a34ead0f37be99bb83808d2b97a5c185 Jul 15 10:47:27.903823 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:27.903794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2fw2b"] Jul 15 10:47:27.906771 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:27.906740 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3439f5a7_5f74_422e_8089_c97186c19f74.slice/crio-dd62a56c4f4d52f80135734602c20e031a2abb5c1ab22978690a958a0c33c685 WatchSource:0}: Error finding container dd62a56c4f4d52f80135734602c20e031a2abb5c1ab22978690a958a0c33c685: Status 404 returned error can't find the container with id dd62a56c4f4d52f80135734602c20e031a2abb5c1ab22978690a958a0c33c685 Jul 15 10:47:28.163801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:28.163720 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mz6kd" Jul 15 10:47:28.200635 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:28.200592 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2fw2b" event={"ID":"3439f5a7-5f74-422e-8089-c97186c19f74","Type":"ContainerStarted","Data":"dd62a56c4f4d52f80135734602c20e031a2abb5c1ab22978690a958a0c33c685"} Jul 15 10:47:28.202117 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:28.202088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mczbg" event={"ID":"0119351c-bc27-4885-b458-ba76a4390657","Type":"ContainerStarted","Data":"155640c10ae90766796c1953d42d17a9a34ead0f37be99bb83808d2b97a5c185"} Jul 15 10:47:28.203219 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:28.203198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d8q55" event={"ID":"4f7beef7-c62f-468d-b02c-2a06d2322773","Type":"ContainerStarted","Data":"4279b095b89db4913c8b94bef28edecd793d4a38e68582af9b304ce257b36337"} Jul 15 10:47:28.610906 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:28.610873 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:29.208538 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:29.208472 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mczbg" event={"ID":"0119351c-bc27-4885-b458-ba76a4390657","Type":"ContainerStarted","Data":"3a0058714de56a9a6cb06406b8e4d294bc41c0025d2dea5e0c60e38a7e90005e"} Jul 15 10:47:30.215087 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.215047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mczbg" event={"ID":"0119351c-bc27-4885-b458-ba76a4390657","Type":"ContainerStarted","Data":"09d41d6c3e701d8d33c596a4df0e7328dc7a50a440c321f69ffa39c899167855"} Jul 15 10:47:30.219973 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.219933 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:30.234055 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.233960 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mczbg" podStartSLOduration=66.112483538 podStartE2EDuration="1m7.233937249s" totalImagesPullingTime="1.121453711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:30.232910021 +0000 UTC m=+68.009361285" watchObservedRunningTime="2026-07-15 10:47:30.233937249 +0000 UTC m=+68.010388491" Jul 15 10:47:30.248651 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.248613 2580 patch_prober.go:28] interesting pod/image-registry-84c8f5d4dd-g7hxg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 10:47:30.248824 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.248674 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" podUID="f48df869-59d0-472d-9b6a-a4df786659ae" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:47:30.320884 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.320836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:30.421876 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.421821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:30.598349 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.598313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:30.690790 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.690754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:30.791136 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.791064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:30.900049 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.900015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:30.924219 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.924184 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88"] Jul 15 10:47:30.927124 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.927102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:30.929646 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.929549 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 10:47:30.929897 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.929851 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 10:47:30.929897 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.929878 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7vdcz\"" Jul 15 10:47:30.930826 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.930776 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 10:47:30.930826 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.930776 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 10:47:30.931048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.930845 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 10:47:30.942310 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.942211 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88"] Jul 15 10:47:30.970873 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.970616 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tnpf5"] Jul 15 10:47:30.972765 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.972739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:30.975568 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.975541 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 10:47:30.975736 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.975542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m8kxz\"" Jul 15 10:47:30.975736 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.975609 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 10:47:30.975926 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.975910 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 10:47:30.987348 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:30.987314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:31.016169 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.016135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.016345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.016175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwlzr\" (UniqueName: \"kubernetes.io/projected/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-kube-api-access-zwlzr\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.016345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.016209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.016345 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.016315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.055288 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.055240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:31.117611 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-wtmp\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.117812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.117812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-root\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.117812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-tls\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.117812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-accelerators-collector-config\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.117812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.117812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-sys\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.118112 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-textfile\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.118112 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:31.117945 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 15 10:47:31.118112 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.117928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.118112 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:31.118024 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-tls podName:a69f4397-2b05-4e80-b5ab-56a8db3d3e2f nodeName:}" failed. No retries permitted until 2026-07-15 10:47:31.618000279 +0000 UTC m=+69.394451494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-mwl88" (UID: "a69f4397-2b05-4e80-b5ab-56a8db3d3e2f") : secret "openshift-state-metrics-tls" not found Jul 15 10:47:31.118112 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.118084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-metrics-client-ca\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.118351 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.118121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwjm\" (UniqueName: \"kubernetes.io/projected/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-kube-api-access-cjwjm\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.118351 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.118152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.118351 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.118183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwlzr\" (UniqueName: \"kubernetes.io/projected/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-kube-api-access-zwlzr\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.118886 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.118859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.120603 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.120557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.127480 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.127441 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwlzr\" (UniqueName: \"kubernetes.io/projected/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-kube-api-access-zwlzr\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.153048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.152952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:31.218870 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.218836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-metrics-client-ca\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.218881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwjm\" (UniqueName: \"kubernetes.io/projected/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-kube-api-access-cjwjm\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.218940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-wtmp\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.218968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-root\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-root\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-wtmp\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-tls\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-accelerators-collector-config\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-sys\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-sys\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219842 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-textfile\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219842 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219842 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-metrics-client-ca\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219842 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-textfile\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.219842 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.219783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-accelerators-collector-config\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.222109 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.221973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-tls\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.222109 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.221989 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2fw2b" event={"ID":"3439f5a7-5f74-422e-8089-c97186c19f74","Type":"ContainerStarted","Data":"bbae411442562f3c8dc66920d0b36d0165d4ac5acaaa4d5652e42cf2fa9529c1"} Jul 15 10:47:31.222317 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.222147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.231096 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.231070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwjm\" (UniqueName: \"kubernetes.io/projected/05b19dc3-60ee-4f6d-b758-8f2f51f6a548-kube-api-access-cjwjm\") pod \"node-exporter-tnpf5\" (UID: \"05b19dc3-60ee-4f6d-b758-8f2f51f6a548\") " pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.239700 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.239645 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2fw2b" podStartSLOduration=65.255798184 podStartE2EDuration="1m8.239626476s" totalImagesPullingTime="2.983828292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:31.238726048 +0000 UTC m=+69.015177340" watchObservedRunningTime="2026-07-15 10:47:31.239626476 +0000 UTC m=+69.016077704" Jul 15 10:47:31.244031 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.243990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:31.282686 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.282645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tnpf5" Jul 15 10:47:31.367354 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.367317 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:31.466621 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.466515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:31.566797 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.566762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:31.623239 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.623195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.625959 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.625923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a69f4397-2b05-4e80-b5ab-56a8db3d3e2f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-mwl88\" (UID: \"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.852143 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.852093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" Jul 15 10:47:31.905444 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:31.905399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:32.018692 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.018656 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 10:47:32.021677 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.021650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.025770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025612 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-cluster-tls-config\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" Jul 15 10:47:32.025770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025631 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-web\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" Jul 15 10:47:32.025770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025612 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-generated\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" Jul 15 10:47:32.025770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025677 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" Jul 15 10:47:32.025770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025694 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls-assets-0\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" Jul 15 10:47:32.025770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025698 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"alertmanager-trusted-ca-bundle\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" Jul 15 10:47:32.025770 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025695 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"alertmanager-main-0\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" podUID="5608afc3-76eb-415c-87ec-c6ec07a25111" pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.026248 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.025984 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-web-config\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" Jul 15 10:47:32.026248 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.026015 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-dockercfg-hwpkr\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hwpkr\"" type="*v1.Secret" Jul 15 10:47:32.026248 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.026046 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-metric\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" Jul 15 10:47:32.026248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.026159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:32.027149 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:32.027116 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls\" is forbidden: User \"system:node:ip-10-0-130-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" Jul 15 10:47:32.049077 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.049034 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 10:47:32.127674 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5608afc3-76eb-415c-87ec-c6ec07a25111-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127674 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-config-volume\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127674 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5608afc3-76eb-415c-87ec-c6ec07a25111-config-out\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5608afc3-76eb-415c-87ec-c6ec07a25111-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5608afc3-76eb-415c-87ec-c6ec07a25111-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztd4n\" (UniqueName: \"kubernetes.io/projected/5608afc3-76eb-415c-87ec-c6ec07a25111-kube-api-access-ztd4n\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5608afc3-76eb-415c-87ec-c6ec07a25111-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.127949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-web-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.128340 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.127973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.128340 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.128018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.141307 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.141281 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84c8f5d4dd-g7hxg" Jul 15 10:47:32.226724 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.226665 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:47:32.229115 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-web-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229253 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229253 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229253 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5608afc3-76eb-415c-87ec-c6ec07a25111-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229405 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-config-volume\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229405 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229405 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5608afc3-76eb-415c-87ec-c6ec07a25111-config-out\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229405 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5608afc3-76eb-415c-87ec-c6ec07a25111-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229639 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5608afc3-76eb-415c-87ec-c6ec07a25111-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229639 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229639 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229789 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd4n\" (UniqueName: \"kubernetes.io/projected/5608afc3-76eb-415c-87ec-c6ec07a25111-kube-api-access-ztd4n\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229789 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5608afc3-76eb-415c-87ec-c6ec07a25111-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.229986 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.229963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5608afc3-76eb-415c-87ec-c6ec07a25111-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.230374 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.230349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5608afc3-76eb-415c-87ec-c6ec07a25111-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.232227 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.232173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5608afc3-76eb-415c-87ec-c6ec07a25111-config-out\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.244844 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.244814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztd4n\" (UniqueName: \"kubernetes.io/projected/5608afc3-76eb-415c-87ec-c6ec07a25111-kube-api-access-ztd4n\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.333175 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.333144 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:32.433803 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.433711 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:32.463950 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:32.463918 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b19dc3_60ee_4f6d_b758_8f2f51f6a548.slice/crio-90db5aca89f55dd9365b9504672a2b96597046e2cbda47d5981e95d35286094f WatchSource:0}: Error finding container 90db5aca89f55dd9365b9504672a2b96597046e2cbda47d5981e95d35286094f: Status 404 returned error can't find the container with id 90db5aca89f55dd9365b9504672a2b96597046e2cbda47d5981e95d35286094f Jul 15 10:47:32.608488 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.608456 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88"] Jul 15 10:47:32.611724 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:32.611689 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69f4397_2b05_4e80_b5ab_56a8db3d3e2f.slice/crio-37da7115a05ae9b6e1ff1dfaf97ba67b50d0d9820665d060a2f17ef5174159f7 WatchSource:0}: Error finding container 37da7115a05ae9b6e1ff1dfaf97ba67b50d0d9820665d060a2f17ef5174159f7: Status 404 returned error can't find the container with id 37da7115a05ae9b6e1ff1dfaf97ba67b50d0d9820665d060a2f17ef5174159f7 Jul 15 10:47:32.621053 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.621023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:32.936278 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.936246 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 10:47:32.942535 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.942449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5608afc3-76eb-415c-87ec-c6ec07a25111-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:32.975342 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.975313 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 10:47:32.979780 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.979744 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:32.980533 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:32.980499 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5608afc3-76eb-415c-87ec-c6ec07a25111-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.008871 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.008832 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 10:47:33.012643 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.012611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.016760 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.016734 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 10:47:33.022483 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.022458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.057749 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.057708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 10:47:33.062398 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.062367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.079823 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.079782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:33.178645 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.178608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:33.212639 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.212536 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 10:47:33.222419 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.222396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-config-volume\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.229416 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:33.229391 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-web-config: failed to sync secret cache: timed out waiting for the condition Jul 15 10:47:33.229850 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:33.229458 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-web-config podName:5608afc3-76eb-415c-87ec-c6ec07a25111 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:33.729442029 +0000 UTC m=+71.505893244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "web-config" (UniqueName: "kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-web-config") pod "alertmanager-main-0" (UID: "5608afc3-76eb-415c-87ec-c6ec07a25111") : failed to sync secret cache: timed out waiting for the condition Jul 15 10:47:33.229850 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:33.229390 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-kube-rbac-proxy-metric: failed to sync secret cache: timed out waiting for the condition Jul 15 10:47:33.229850 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:33.229529 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-metric podName:5608afc3-76eb-415c-87ec-c6ec07a25111 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:33.729510102 +0000 UTC m=+71.505961317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-kube-rbac-proxy-metric" (UniqueName: "kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-metric") pod "alertmanager-main-0" (UID: "5608afc3-76eb-415c-87ec-c6ec07a25111") : failed to sync secret cache: timed out waiting for the condition Jul 15 10:47:33.230522 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:33.230501 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-cluster-tls-config: failed to sync secret cache: timed out waiting for the condition Jul 15 10:47:33.230645 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:47:33.230560 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-cluster-tls-config podName:5608afc3-76eb-415c-87ec-c6ec07a25111 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:33.730542464 +0000 UTC m=+71.506993688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-tls-config" (UniqueName: "kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-cluster-tls-config") pod "alertmanager-main-0" (UID: "5608afc3-76eb-415c-87ec-c6ec07a25111") : failed to sync secret cache: timed out waiting for the condition Jul 15 10:47:33.231990 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.231960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnpf5" event={"ID":"05b19dc3-60ee-4f6d-b758-8f2f51f6a548","Type":"ContainerStarted","Data":"90db5aca89f55dd9365b9504672a2b96597046e2cbda47d5981e95d35286094f"} Jul 15 10:47:33.233329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.233298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d8q55" event={"ID":"4f7beef7-c62f-468d-b02c-2a06d2322773","Type":"ContainerStarted","Data":"bcacc4c5f5e6ed56f146d5f05984c925327bd3d3f9f01e9f912dd3fc2c12d9e2"} Jul 15 10:47:33.235213 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.235189 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" event={"ID":"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f","Type":"ContainerStarted","Data":"57f34122fda746130f825ae99cc4875f5c50941b7d989bc68f9db0e0809449af"} Jul 15 10:47:33.235324 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.235246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" event={"ID":"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f","Type":"ContainerStarted","Data":"c680334c557e1635699f87925d7c8232c1cf00e5fe2eeee95f1df81b70c6962a"} Jul 15 10:47:33.235324 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.235274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" event={"ID":"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f","Type":"ContainerStarted","Data":"37da7115a05ae9b6e1ff1dfaf97ba67b50d0d9820665d060a2f17ef5174159f7"} Jul 15 10:47:33.235789 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.235772 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 10:47:33.245651 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.245621 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 10:47:33.252753 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.252694 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d8q55" podStartSLOduration=65.522867283 podStartE2EDuration="1m10.252677824s" totalImagesPullingTime="4.729810541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:33.251068358 +0000 UTC m=+71.027519596" watchObservedRunningTime="2026-07-15 10:47:33.252677824 +0000 UTC m=+71.029129058" Jul 15 10:47:33.577195 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.577115 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 10:47:33.582108 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.582083 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hwpkr\"" Jul 15 10:47:33.744531 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.744493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-web-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.744726 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.744546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.744726 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.744630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.746952 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.746924 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.747095 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.747074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-web-config\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.747134 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.747074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5608afc3-76eb-415c-87ec-c6ec07a25111-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5608afc3-76eb-415c-87ec-c6ec07a25111\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.833523 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.833426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 10:47:33.893728 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.893699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:33.993629 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:33.993587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:34.241170 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.241103 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:34.336051 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.336025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 10:47:34.380910 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:34.380865 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5608afc3_76eb_415c_87ec_c6ec07a25111.slice/crio-c26fec900db57da8a396d3ec3c1b22cdd94d079c93d159daea3dcd70cbe14105 WatchSource:0}: Error finding container c26fec900db57da8a396d3ec3c1b22cdd94d079c93d159daea3dcd70cbe14105: Status 404 returned error can't find the container with id c26fec900db57da8a396d3ec3c1b22cdd94d079c93d159daea3dcd70cbe14105 Jul 15 10:47:34.501739 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.501700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:34.975232 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.975199 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-578f4467db-f5ltg"] Jul 15 10:47:34.977708 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.977683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:34.981055 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.981013 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 10:47:34.981204 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.981181 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 10:47:34.981320 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.981302 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 10:47:34.983726 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.983699 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 10:47:34.983726 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.983708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 10:47:34.983969 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.983955 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bpcqp\"" Jul 15 10:47:34.984038 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:34.983986 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bgstjmb395sdv\"" Jul 15 10:47:35.010640 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.010610 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578f4467db-f5ltg"] Jul 15 10:47:35.054477 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpph\" (UniqueName: \"kubernetes.io/projected/614d32c4-4771-4134-9ef9-ce3233bbe6d5-kube-api-access-xkpph\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.054477 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.054758 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-tls\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.054758 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/614d32c4-4771-4134-9ef9-ce3233bbe6d5-metrics-client-ca\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.054758 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-grpc-tls\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.054758 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.054963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.054963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.054848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.155902 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.155858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-grpc-tls\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.155916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.155946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.155985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.156019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpph\" (UniqueName: \"kubernetes.io/projected/614d32c4-4771-4134-9ef9-ce3233bbe6d5-kube-api-access-xkpph\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.156052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.156091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-tls\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156164 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.156126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/614d32c4-4771-4134-9ef9-ce3233bbe6d5-metrics-client-ca\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.156908 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.156885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/614d32c4-4771-4134-9ef9-ce3233bbe6d5-metrics-client-ca\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.159222 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.159190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.160046 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.159626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.160046 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.160013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.160237 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.160084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-tls\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.160237 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.160131 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-grpc-tls\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.160583 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.160541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/614d32c4-4771-4134-9ef9-ce3233bbe6d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.166487 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.166457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpph\" (UniqueName: \"kubernetes.io/projected/614d32c4-4771-4134-9ef9-ce3233bbe6d5-kube-api-access-xkpph\") pod \"thanos-querier-578f4467db-f5ltg\" (UID: \"614d32c4-4771-4134-9ef9-ce3233bbe6d5\") " pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.241871 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.241826 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4"] Jul 15 10:47:35.245196 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.245169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.246827 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.246794 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" event={"ID":"a69f4397-2b05-4e80-b5ab-56a8db3d3e2f","Type":"ContainerStarted","Data":"b2f2c9dc6111e25221129a59ef077cb9e27f98b1f5fed0e8b6d652b04803e118"} Jul 15 10:47:35.247678 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.247653 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 10:47:35.247798 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.247743 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 10:47:35.248226 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.247948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-49mrm913oobkh\"" Jul 15 10:47:35.248226 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.247989 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 10:47:35.248226 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.247999 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gztcl\"" Jul 15 10:47:35.248226 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.248000 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 10:47:35.249030 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.249003 2580 generic.go:356] "Generic (PLEG): container finished" podID="05b19dc3-60ee-4f6d-b758-8f2f51f6a548" containerID="e81a353eaeec878572bd415d13115a748b0a9ededacb7bd43246726aba26a138" exitCode=0 Jul 15 10:47:35.249143 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.249097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnpf5" event={"ID":"05b19dc3-60ee-4f6d-b758-8f2f51f6a548","Type":"ContainerDied","Data":"e81a353eaeec878572bd415d13115a748b0a9ededacb7bd43246726aba26a138"} Jul 15 10:47:35.250433 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.250410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerStarted","Data":"c26fec900db57da8a396d3ec3c1b22cdd94d079c93d159daea3dcd70cbe14105"} Jul 15 10:47:35.257678 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.257644 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4"] Jul 15 10:47:35.292490 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.292391 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:35.312943 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.312882 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-mwl88" podStartSLOduration=3.704627786 podStartE2EDuration="5.312862183s" totalImagesPullingTime="1.608234397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:35.312365452 +0000 UTC m=+73.088816691" watchObservedRunningTime="2026-07-15 10:47:35.312862183 +0000 UTC m=+73.089313420" Jul 15 10:47:35.357702 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.357673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3956c90-8880-4430-a6af-d63809e5e7a5-audit-log\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.357802 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.357734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-client-ca-bundle\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.357864 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.357812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-secret-metrics-server-client-certs\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.357919 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.357867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-secret-metrics-server-tls\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.357919 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.357908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3956c90-8880-4430-a6af-d63809e5e7a5-metrics-server-audit-profiles\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.358022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.357932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3956c90-8880-4430-a6af-d63809e5e7a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.358073 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.358036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455r7\" (UniqueName: \"kubernetes.io/projected/a3956c90-8880-4430-a6af-d63809e5e7a5-kube-api-access-455r7\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.458879 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.458847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455r7\" (UniqueName: \"kubernetes.io/projected/a3956c90-8880-4430-a6af-d63809e5e7a5-kube-api-access-455r7\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.458895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3956c90-8880-4430-a6af-d63809e5e7a5-audit-log\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.458931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-client-ca-bundle\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.458960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-secret-metrics-server-client-certs\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.458999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-secret-metrics-server-tls\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459238 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.459038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3956c90-8880-4430-a6af-d63809e5e7a5-metrics-server-audit-profiles\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459238 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.459063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3956c90-8880-4430-a6af-d63809e5e7a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459634 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.459519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3956c90-8880-4430-a6af-d63809e5e7a5-audit-log\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.459963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.459934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3956c90-8880-4430-a6af-d63809e5e7a5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.460294 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.460273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3956c90-8880-4430-a6af-d63809e5e7a5-metrics-server-audit-profiles\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.461484 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.461462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-secret-metrics-server-client-certs\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.461567 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.461532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-client-ca-bundle\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.461567 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.461565 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3956c90-8880-4430-a6af-d63809e5e7a5-secret-metrics-server-tls\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.468307 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.468281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-455r7\" (UniqueName: \"kubernetes.io/projected/a3956c90-8880-4430-a6af-d63809e5e7a5-kube-api-access-455r7\") pod \"metrics-server-5b6f74dfbc-lm4p4\" (UID: \"a3956c90-8880-4430-a6af-d63809e5e7a5\") " pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.478191 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.478161 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578f4467db-f5ltg"] Jul 15 10:47:35.497958 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:35.497933 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614d32c4_4771_4134_9ef9_ce3233bbe6d5.slice/crio-e33148b2e160a7cb04d15fcdc8aeaa8381e02ca5f61d9e2cb04d4a6163e372cd WatchSource:0}: Error finding container e33148b2e160a7cb04d15fcdc8aeaa8381e02ca5f61d9e2cb04d4a6163e372cd: Status 404 returned error can't find the container with id e33148b2e160a7cb04d15fcdc8aeaa8381e02ca5f61d9e2cb04d4a6163e372cd Jul 15 10:47:35.557016 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.556980 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:35.711153 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.711055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:35.715759 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.715729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4"] Jul 15 10:47:35.718531 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:35.718501 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3956c90_8880_4430_a6af_d63809e5e7a5.slice/crio-c0719a0e18b359f339cbb52d887209d97a1857a80e361d10bb2d9d3f9871231c WatchSource:0}: Error finding container c0719a0e18b359f339cbb52d887209d97a1857a80e361d10bb2d9d3f9871231c: Status 404 returned error can't find the container with id c0719a0e18b359f339cbb52d887209d97a1857a80e361d10bb2d9d3f9871231c Jul 15 10:47:35.792676 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:35.792639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:36.215432 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.215399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:36.257280 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.257244 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnpf5" event={"ID":"05b19dc3-60ee-4f6d-b758-8f2f51f6a548","Type":"ContainerStarted","Data":"d2770690513a413d9df17846f972866d45cb1ba77324a478ff9ff90edce1399d"} Jul 15 10:47:36.257746 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.257309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tnpf5" event={"ID":"05b19dc3-60ee-4f6d-b758-8f2f51f6a548","Type":"ContainerStarted","Data":"53953476a1a362c306acbf7d4d017136c424bb4c190f83d1b423ef48698b8a56"} Jul 15 10:47:36.259020 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.258992 2580 generic.go:356] "Generic (PLEG): container finished" podID="5608afc3-76eb-415c-87ec-c6ec07a25111" containerID="3f2832e2459d3df7e1e882d78953478a9ad322b9f42e2ddcf475ca7cdd27b1e5" exitCode=0 Jul 15 10:47:36.259141 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.259083 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerDied","Data":"3f2832e2459d3df7e1e882d78953478a9ad322b9f42e2ddcf475ca7cdd27b1e5"} Jul 15 10:47:36.260628 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.260384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" event={"ID":"a3956c90-8880-4430-a6af-d63809e5e7a5","Type":"ContainerStarted","Data":"c0719a0e18b359f339cbb52d887209d97a1857a80e361d10bb2d9d3f9871231c"} Jul 15 10:47:36.261733 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.261706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" event={"ID":"614d32c4-4771-4134-9ef9-ce3233bbe6d5","Type":"ContainerStarted","Data":"e33148b2e160a7cb04d15fcdc8aeaa8381e02ca5f61d9e2cb04d4a6163e372cd"} Jul 15 10:47:36.297724 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.297683 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85cb8458ff-6ws4t"] Jul 15 10:47:36.300712 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.300692 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.304084 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.304063 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 10:47:36.304513 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.304494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 10:47:36.304513 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.304506 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 10:47:36.304676 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.304497 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 10:47:36.304909 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.304876 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 10:47:36.305153 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.305130 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7vc54\"" Jul 15 10:47:36.314055 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.313237 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 10:47:36.319329 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.319265 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tnpf5" podStartSLOduration=4.48410859 podStartE2EDuration="6.319243604s" totalImagesPullingTime="1.728793768s" totalInitContainerRuntime="106.341246ms" isStatefulPod=true podCreationTimestamp="2026-07-15 10:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:36.318027747 +0000 UTC m=+74.094479010" watchObservedRunningTime="2026-07-15 10:47:36.319243604 +0000 UTC m=+74.095694842" Jul 15 10:47:36.327704 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.327666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85cb8458ff-6ws4t"] Jul 15 10:47:36.473410 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-federate-client-tls\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.473410 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-secret-telemeter-client\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.473410 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-serving-certs-ca-bundle\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.473766 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmwv5\" (UniqueName: \"kubernetes.io/projected/70c6f363-2fd5-4643-a730-24519af9b8c0-kube-api-access-dmwv5\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.473766 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.473766 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.473766 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-metrics-client-ca\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.473766 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.473740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-telemeter-client-tls\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-metrics-client-ca\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-telemeter-client-tls\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-federate-client-tls\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-secret-telemeter-client\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-serving-certs-ca-bundle\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmwv5\" (UniqueName: \"kubernetes.io/projected/70c6f363-2fd5-4643-a730-24519af9b8c0-kube-api-access-dmwv5\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.574963 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.574948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.575610 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.575533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-metrics-client-ca\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.575754 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.575723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-serving-certs-ca-bundle\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.575952 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.575908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c6f363-2fd5-4643-a730-24519af9b8c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.578060 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.578011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-federate-client-tls\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.578312 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.578289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-telemeter-client-tls\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.578991 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.578963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.579198 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.579172 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/70c6f363-2fd5-4643-a730-24519af9b8c0-secret-telemeter-client\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.590032 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.589995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmwv5\" (UniqueName: \"kubernetes.io/projected/70c6f363-2fd5-4643-a730-24519af9b8c0-kube-api-access-dmwv5\") pod \"telemeter-client-85cb8458ff-6ws4t\" (UID: \"70c6f363-2fd5-4643-a730-24519af9b8c0\") " pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.609877 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.609841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" Jul 15 10:47:36.698717 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.698679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:36.792368 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:36.792259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:37.231119 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:37.231080 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85cb8458ff-6ws4t"] Jul 15 10:47:37.268295 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:37.268218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" event={"ID":"a3956c90-8880-4430-a6af-d63809e5e7a5","Type":"ContainerStarted","Data":"0a4413470da1077f0debf67fc3c7c53d1e2d0d0dffb6649a58fa0660f6bded7b"} Jul 15 10:47:37.300264 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:37.300206 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" podStartSLOduration=0.907566012 podStartE2EDuration="2.300181098s" totalImagesPullingTime="1.392615086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:37.299096759 +0000 UTC m=+75.075548000" watchObservedRunningTime="2026-07-15 10:47:37.300181098 +0000 UTC m=+75.076632336" Jul 15 10:47:37.356314 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:37.356274 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:37.457162 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:37.457117 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:37.805072 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:47:37.804995 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c6f363_2fd5_4643_a730_24519af9b8c0.slice/crio-988c2c811d41e23ce97df8bfe2a309dfdef215ad805eca1ec2a71e996ff84776 WatchSource:0}: Error finding container 988c2c811d41e23ce97df8bfe2a309dfdef215ad805eca1ec2a71e996ff84776: Status 404 returned error can't find the container with id 988c2c811d41e23ce97df8bfe2a309dfdef215ad805eca1ec2a71e996ff84776 Jul 15 10:47:38.277321 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:38.277292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" event={"ID":"70c6f363-2fd5-4643-a730-24519af9b8c0","Type":"ContainerStarted","Data":"988c2c811d41e23ce97df8bfe2a309dfdef215ad805eca1ec2a71e996ff84776"} Jul 15 10:47:38.324276 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:38.324183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:39.227398 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.227364 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:39.284415 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.284380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" event={"ID":"614d32c4-4771-4134-9ef9-ce3233bbe6d5","Type":"ContainerStarted","Data":"83b80d3becc464f44e53689195ec37acee727bca3a68b948e593ac73da0d6203"} Jul 15 10:47:39.284915 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.284450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" event={"ID":"614d32c4-4771-4134-9ef9-ce3233bbe6d5","Type":"ContainerStarted","Data":"72d345ebe9d4fcf34922ae14f448ceb5bd7c0f3ccd09c95af56aa43489c5e2be"} Jul 15 10:47:39.284915 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.284482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" event={"ID":"614d32c4-4771-4134-9ef9-ce3233bbe6d5","Type":"ContainerStarted","Data":"bd1e23077b8855f2724b5cfa7fef2903f894bd032ed12fad8af1de3fae939c18"} Jul 15 10:47:39.287421 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.287384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerStarted","Data":"dd5e33bad071d8b289334c13cf2733e2d15fc56fa7ac04db422de1479bb4e0d2"} Jul 15 10:47:39.287597 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.287460 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerStarted","Data":"e23f646f6bfb014a8c079b6f78e03cfb48d6909e3d44bd9c00c50cef35e2fc78"} Jul 15 10:47:39.287597 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.287510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerStarted","Data":"ccdc6fe2c283cefda144738bc213cc635aba0b3d4d1a4285309f03a1b543d10b"} Jul 15 10:47:39.287597 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.287549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerStarted","Data":"8ac48267b19eb8b2508ce1b2a5e7caf9e179336cfcab999f97990bcbcf9f7547"} Jul 15 10:47:39.287765 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.287606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerStarted","Data":"cbad941e1a593b7298cc117de19f38ccf7bc2351ff82b03336b6aed2ede1d935"} Jul 15 10:47:39.321496 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:39.321456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:40.040066 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.039980 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:40.113696 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.113663 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:40.294824 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.294729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5608afc3-76eb-415c-87ec-c6ec07a25111","Type":"ContainerStarted","Data":"ac374cd0c17295f359d803be1bad3f855a9467c31089a65345e2e21e133da09b"} Jul 15 10:47:40.298324 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.298286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" event={"ID":"614d32c4-4771-4134-9ef9-ce3233bbe6d5","Type":"ContainerStarted","Data":"789ac6ea291bc75bbee59b8631758a05abf3147b1ea85f8ce34f8a7868206211"} Jul 15 10:47:40.298750 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.298367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" event={"ID":"614d32c4-4771-4134-9ef9-ce3233bbe6d5","Type":"ContainerStarted","Data":"f935c8a72507b711270d0f9b993f5f2dd49a7b166cafd89ed2fa168fd8072583"} Jul 15 10:47:40.298750 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.298412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" event={"ID":"614d32c4-4771-4134-9ef9-ce3233bbe6d5","Type":"ContainerStarted","Data":"85b0053cb23828aa751763bfa8c5f0808d5b0c205e56926c1d5c689ad310ffd9"} Jul 15 10:47:40.298750 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.298456 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:40.300658 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.300633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" event={"ID":"70c6f363-2fd5-4643-a730-24519af9b8c0","Type":"ContainerStarted","Data":"e034bb732c32ed22667ab5e1991551a5fe53cb6ace02b0571ec2e56585819c8a"} Jul 15 10:47:40.300779 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.300683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" event={"ID":"70c6f363-2fd5-4643-a730-24519af9b8c0","Type":"ContainerStarted","Data":"0914d64d3d9170e121fae02229e9b08dc84db9ea85a22f085eec4c36f59d471b"} Jul 15 10:47:40.300779 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.300702 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" event={"ID":"70c6f363-2fd5-4643-a730-24519af9b8c0","Type":"ContainerStarted","Data":"fa371e97e2d64da895201a773d5d730d18bc6379ae27246682859d04d47fca35"} Jul 15 10:47:40.334673 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.334617 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.202374799 podStartE2EDuration="9.33459989s" totalImagesPullingTime="4.109454573s" totalInitContainerRuntime="22.770518ms" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:40.332417778 +0000 UTC m=+78.108869015" watchObservedRunningTime="2026-07-15 10:47:40.33459989 +0000 UTC m=+78.111051127" Jul 15 10:47:40.373257 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:40.373203 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" podStartSLOduration=2.324071252 podStartE2EDuration="6.373181966s" totalImagesPullingTime="4.049110714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:40.371833382 +0000 UTC m=+78.148284618" watchObservedRunningTime="2026-07-15 10:47:40.373181966 +0000 UTC m=+78.149633207" Jul 15 10:47:42.308407 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:42.308290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:42.412672 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:42.412637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:46.310227 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:46.310200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-578f4467db-f5ltg" Jul 15 10:47:46.352773 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:46.352721 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85cb8458ff-6ws4t" podStartSLOduration=8.397287927 podStartE2EDuration="10.352702848s" totalImagesPullingTime="1.955414921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:40.414433218 +0000 UTC m=+78.190884483" watchObservedRunningTime="2026-07-15 10:47:46.352702848 +0000 UTC m=+84.129154085" Jul 15 10:47:46.393124 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:46.393066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:46.489834 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:46.489796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:47.465411 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:47.465371 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:47.837044 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:47.837011 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:47.922650 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:47.922608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:51.442672 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:51.442636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:51.513401 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:51.513371 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:52.470158 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:52.470100 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:52.572198 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:52.572170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:52.627590 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:52.627552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:47:55.557460 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:55.557431 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:47:55.557903 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:47:55.557488 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:48:03.238280 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:03.238244 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2fw2b" Jul 15 10:48:10.026887 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:10.026851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:10.088661 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:10.088629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:15.563157 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:15.563126 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:48:15.567022 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:15.566992 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b6f74dfbc-lm4p4" Jul 15 10:48:25.615709 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:25.615669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:25.689178 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:25.689150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:47.195862 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:47.195799 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:47.266289 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:47.266254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:59.107856 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:59.107793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:48:59.654097 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:48:59.654055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:01.506023 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:01.505989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:01.599325 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:01.599275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:01.661817 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:01.661780 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:02.253828 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:02.253792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:10.037553 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:10.037518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:10.134458 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:10.134425 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:10.549197 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:10.549157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:12.840904 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:12.840811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:12.926690 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:12.926652 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:17.528387 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:17.528350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:28.750877 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:28.750840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:29.337773 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:29.337738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:45.849443 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:45.849401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:57.798460 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:57.798424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:57.897652 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:57.897616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:58.231696 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:58.231657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:58.327981 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:58.327945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:58.430375 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:58.430306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:49:59.204006 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.203969 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6"] Jul 15 10:49:59.206258 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.206239 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.209209 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.209182 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 10:49:59.209355 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.209182 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mp6pj\"" Jul 15 10:49:59.210034 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.210013 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 10:49:59.219864 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.219830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6"] Jul 15 10:49:59.256222 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.256172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.256407 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.256264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.256407 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.256293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c888h\" (UniqueName: \"kubernetes.io/projected/9d31e7eb-ab21-4930-9580-1edfd592b6cc-kube-api-access-c888h\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.357006 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.356957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.357184 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.357015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c888h\" (UniqueName: \"kubernetes.io/projected/9d31e7eb-ab21-4930-9580-1edfd592b6cc-kube-api-access-c888h\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.357184 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.357066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.357328 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.357308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.357381 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.357357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.367257 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.367222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c888h\" (UniqueName: \"kubernetes.io/projected/9d31e7eb-ab21-4930-9580-1edfd592b6cc-kube-api-access-c888h\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.515470 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.515366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:49:59.654361 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.654325 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6"] Jul 15 10:49:59.657260 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:49:59.657228 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d31e7eb_ab21_4930_9580_1edfd592b6cc.slice/crio-fb2df81e8c2aa538728e0c01160c8698b8ba28c0f1c4f9d9ceba428800e1b87d WatchSource:0}: Error finding container fb2df81e8c2aa538728e0c01160c8698b8ba28c0f1c4f9d9ceba428800e1b87d: Status 404 returned error can't find the container with id fb2df81e8c2aa538728e0c01160c8698b8ba28c0f1c4f9d9ceba428800e1b87d Jul 15 10:49:59.838039 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:49:59.838011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" event={"ID":"9d31e7eb-ab21-4930-9580-1edfd592b6cc","Type":"ContainerStarted","Data":"fb2df81e8c2aa538728e0c01160c8698b8ba28c0f1c4f9d9ceba428800e1b87d"} Jul 15 10:50:06.867824 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:06.867783 2580 generic.go:356] "Generic (PLEG): container finished" podID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerID="8107e94557af4b8e150af2a170ded0f0057a14a05861adaf0199c5d82fb67a13" exitCode=0 Jul 15 10:50:06.868232 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:06.867836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" event={"ID":"9d31e7eb-ab21-4930-9580-1edfd592b6cc","Type":"ContainerDied","Data":"8107e94557af4b8e150af2a170ded0f0057a14a05861adaf0199c5d82fb67a13"} Jul 15 10:50:08.877194 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:08.877151 2580 generic.go:356] "Generic (PLEG): container finished" podID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerID="1660ed692bb0d6b253e78d002faa998fec42bcc3e5432cd0b95df6c83f5108bd" exitCode=0 Jul 15 10:50:08.877650 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:08.877210 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" event={"ID":"9d31e7eb-ab21-4930-9580-1edfd592b6cc","Type":"ContainerDied","Data":"1660ed692bb0d6b253e78d002faa998fec42bcc3e5432cd0b95df6c83f5108bd"} Jul 15 10:50:15.908727 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:15.908685 2580 generic.go:356] "Generic (PLEG): container finished" podID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerID="655ae05bf24a569373c251c1eefcc5f76a6ccf44ab527be68978294ba488ca9d" exitCode=0 Jul 15 10:50:15.909140 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:15.908763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" event={"ID":"9d31e7eb-ab21-4930-9580-1edfd592b6cc","Type":"ContainerDied","Data":"655ae05bf24a569373c251c1eefcc5f76a6ccf44ab527be68978294ba488ca9d"} Jul 15 10:50:17.044460 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.044433 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:50:17.218027 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.217929 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-util\") pod \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " Jul 15 10:50:17.218027 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.217974 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-bundle\") pod \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " Jul 15 10:50:17.218027 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.218017 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c888h\" (UniqueName: \"kubernetes.io/projected/9d31e7eb-ab21-4930-9580-1edfd592b6cc-kube-api-access-c888h\") pod \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\" (UID: \"9d31e7eb-ab21-4930-9580-1edfd592b6cc\") " Jul 15 10:50:17.218714 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.218687 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-bundle" (OuterVolumeSpecName: "bundle") pod "9d31e7eb-ab21-4930-9580-1edfd592b6cc" (UID: "9d31e7eb-ab21-4930-9580-1edfd592b6cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:50:17.220239 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.220216 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d31e7eb-ab21-4930-9580-1edfd592b6cc-kube-api-access-c888h" (OuterVolumeSpecName: "kube-api-access-c888h") pod "9d31e7eb-ab21-4930-9580-1edfd592b6cc" (UID: "9d31e7eb-ab21-4930-9580-1edfd592b6cc"). InnerVolumeSpecName "kube-api-access-c888h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:50:17.223692 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.223659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-util" (OuterVolumeSpecName: "util") pod "9d31e7eb-ab21-4930-9580-1edfd592b6cc" (UID: "9d31e7eb-ab21-4930-9580-1edfd592b6cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 10:50:17.318888 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.318846 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c888h\" (UniqueName: \"kubernetes.io/projected/9d31e7eb-ab21-4930-9580-1edfd592b6cc-kube-api-access-c888h\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:50:17.318888 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.318879 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-util\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:50:17.318888 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.318888 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d31e7eb-ab21-4930-9580-1edfd592b6cc-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:50:17.918100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.918056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" event={"ID":"9d31e7eb-ab21-4930-9580-1edfd592b6cc","Type":"ContainerDied","Data":"fb2df81e8c2aa538728e0c01160c8698b8ba28c0f1c4f9d9ceba428800e1b87d"} Jul 15 10:50:17.918100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.918085 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mz8r6" Jul 15 10:50:17.918358 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:17.918112 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2df81e8c2aa538728e0c01160c8698b8ba28c0f1c4f9d9ceba428800e1b87d" Jul 15 10:50:20.210010 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:20.209970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:20.307553 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:20.307522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:21.871172 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871131 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd"] Jul 15 10:50:21.871793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871563 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerName="extract" Jul 15 10:50:21.871793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871600 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerName="extract" Jul 15 10:50:21.871793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871615 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerName="pull" Jul 15 10:50:21.871793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871623 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerName="pull" Jul 15 10:50:21.871793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871639 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerName="util" Jul 15 10:50:21.871793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871647 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerName="util" Jul 15 10:50:21.871793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.871733 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d31e7eb-ab21-4930-9580-1edfd592b6cc" containerName="extract" Jul 15 10:50:21.875066 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.875043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:21.878809 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.878778 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 15 10:50:21.878957 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.878882 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-ss844\"" Jul 15 10:50:21.879229 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.879213 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 15 10:50:21.879400 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.879377 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 15 10:50:21.892734 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:21.892701 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd"] Jul 15 10:50:22.062108 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.062061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmwx\" (UniqueName: \"kubernetes.io/projected/7e7ece8d-8dd6-4f54-b0e2-191e0b011c09-kube-api-access-xcmwx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd\" (UID: \"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:22.062294 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.062134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7e7ece8d-8dd6-4f54-b0e2-191e0b011c09-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd\" (UID: \"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:22.162629 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.162530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7e7ece8d-8dd6-4f54-b0e2-191e0b011c09-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd\" (UID: \"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:22.162761 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.162630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmwx\" (UniqueName: \"kubernetes.io/projected/7e7ece8d-8dd6-4f54-b0e2-191e0b011c09-kube-api-access-xcmwx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd\" (UID: \"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:22.164968 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.164943 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7e7ece8d-8dd6-4f54-b0e2-191e0b011c09-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd\" (UID: \"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:22.172376 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.172351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmwx\" (UniqueName: \"kubernetes.io/projected/7e7ece8d-8dd6-4f54-b0e2-191e0b011c09-kube-api-access-xcmwx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd\" (UID: \"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:22.184793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.184764 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:22.321323 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.321287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd"] Jul 15 10:50:22.325696 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:50:22.325656 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7ece8d_8dd6_4f54_b0e2_191e0b011c09.slice/crio-da11004e1e7ac9bf120cce4a0f8fb70ac9681de0094ec33fff6f7c22dd5a91b1 WatchSource:0}: Error finding container da11004e1e7ac9bf120cce4a0f8fb70ac9681de0094ec33fff6f7c22dd5a91b1: Status 404 returned error can't find the container with id da11004e1e7ac9bf120cce4a0f8fb70ac9681de0094ec33fff6f7c22dd5a91b1 Jul 15 10:50:22.944057 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:22.944015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" event={"ID":"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09","Type":"ContainerStarted","Data":"da11004e1e7ac9bf120cce4a0f8fb70ac9681de0094ec33fff6f7c22dd5a91b1"} Jul 15 10:50:25.700761 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:25.700726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:25.959147 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:25.959055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" event={"ID":"7e7ece8d-8dd6-4f54-b0e2-191e0b011c09","Type":"ContainerStarted","Data":"8b0407a21162bb00e6a12be10f7498ea7fb99c919a5a895e703dc1db36986d0d"} Jul 15 10:50:25.959310 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:25.959204 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:25.984075 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:25.984021 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" podStartSLOduration=2.138537885 podStartE2EDuration="4.984002865s" totalImagesPullingTime="2.84546498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:50:25.982724723 +0000 UTC m=+243.759175961" watchObservedRunningTime="2026-07-15 10:50:25.984002865 +0000 UTC m=+243.760454102" Jul 15 10:50:25.986127 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:25.986098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.007568 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.007536 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-4fq6d"] Jul 15 10:50:26.010759 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.010741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.013563 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.013539 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 15 10:50:26.013686 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.013605 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 15 10:50:26.013755 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.013686 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9g8nr\"" Jul 15 10:50:26.020729 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.020704 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-4fq6d"] Jul 15 10:50:26.095620 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.095564 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdn2j\" (UniqueName: \"kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-kube-api-access-bdn2j\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.095620 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.095624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/73313f97-ee06-4f0e-8153-95cce511bc3a-cabundle0\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.095932 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.095722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-certificates\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.169565 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.169522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.196718 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.196680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-certificates\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.196875 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.196732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdn2j\" (UniqueName: \"kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-kube-api-access-bdn2j\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.196875 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.196826 2580 secret.go:281] references non-existent secret key: ca.crt Jul 15 10:50:26.196875 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.196847 2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 15 10:50:26.196875 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.196856 2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-4fq6d: references non-existent secret key: ca.crt Jul 15 10:50:26.196875 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.196852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/73313f97-ee06-4f0e-8153-95cce511bc3a-cabundle0\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.197062 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.196903 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-certificates podName:73313f97-ee06-4f0e-8153-95cce511bc3a nodeName:}" failed. No retries permitted until 2026-07-15 10:50:26.696887087 +0000 UTC m=+244.473338302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-certificates") pod "keda-operator-77f648d694-4fq6d" (UID: "73313f97-ee06-4f0e-8153-95cce511bc3a") : references non-existent secret key: ca.crt Jul 15 10:50:26.197450 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.197431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/73313f97-ee06-4f0e-8153-95cce511bc3a-cabundle0\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.206920 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.206890 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdn2j\" (UniqueName: \"kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-kube-api-access-bdn2j\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.212710 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.212638 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-bkx86"] Jul 15 10:50:26.215949 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.215931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.218403 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.218381 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 15 10:50:26.225885 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.225858 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-bkx86"] Jul 15 10:50:26.231809 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.231765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.297450 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.297401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-certificates\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.297635 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.297480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9b706db2-ae11-4582-8cde-0862445ba072-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.297635 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.297518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfg7h\" (UniqueName: \"kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-kube-api-access-kfg7h\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.398529 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.398471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfg7h\" (UniqueName: \"kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-kube-api-access-kfg7h\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.398734 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.398600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-certificates\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.398734 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.398651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9b706db2-ae11-4582-8cde-0862445ba072-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.398734 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.398709 2580 secret.go:281] references non-existent secret key: tls.crt Jul 15 10:50:26.398734 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.398733 2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Jul 15 10:50:26.398877 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.398756 2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-bkx86: references non-existent secret key: tls.crt Jul 15 10:50:26.398877 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:50:26.398827 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-certificates podName:9b706db2-ae11-4582-8cde-0862445ba072 nodeName:}" failed. No retries permitted until 2026-07-15 10:50:26.898809093 +0000 UTC m=+244.675260312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-certificates") pod "keda-metrics-apiserver-55557df8c-bkx86" (UID: "9b706db2-ae11-4582-8cde-0862445ba072") : references non-existent secret key: tls.crt Jul 15 10:50:26.399009 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.398993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9b706db2-ae11-4582-8cde-0862445ba072-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.409868 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.409841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfg7h\" (UniqueName: \"kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-kube-api-access-kfg7h\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.437598 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.437541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.592772 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.592737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.622114 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.619795 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-f5j2z"] Jul 15 10:50:26.624421 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.624364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.628556 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.628530 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 15 10:50:26.640809 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.640774 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-f5j2z"] Jul 15 10:50:26.696234 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.696203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.701422 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.701394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/386123cf-4441-4cbf-962e-8c4c3e03dec1-certificates\") pod \"keda-admission-56f9798bb4-f5j2z\" (UID: \"386123cf-4441-4cbf-962e-8c4c3e03dec1\") " pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.701724 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.701461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dh9\" (UniqueName: \"kubernetes.io/projected/386123cf-4441-4cbf-962e-8c4c3e03dec1-kube-api-access-m8dh9\") pod \"keda-admission-56f9798bb4-f5j2z\" (UID: \"386123cf-4441-4cbf-962e-8c4c3e03dec1\") " pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.701724 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.701520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-certificates\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.704012 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.703979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/73313f97-ee06-4f0e-8153-95cce511bc3a-certificates\") pod \"keda-operator-77f648d694-4fq6d\" (UID: \"73313f97-ee06-4f0e-8153-95cce511bc3a\") " pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.798546 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.798513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.802847 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.802819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/386123cf-4441-4cbf-962e-8c4c3e03dec1-certificates\") pod \"keda-admission-56f9798bb4-f5j2z\" (UID: \"386123cf-4441-4cbf-962e-8c4c3e03dec1\") " pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.802988 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.802890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dh9\" (UniqueName: \"kubernetes.io/projected/386123cf-4441-4cbf-962e-8c4c3e03dec1-kube-api-access-m8dh9\") pod \"keda-admission-56f9798bb4-f5j2z\" (UID: \"386123cf-4441-4cbf-962e-8c4c3e03dec1\") " pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.805337 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.805303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/386123cf-4441-4cbf-962e-8c4c3e03dec1-certificates\") pod \"keda-admission-56f9798bb4-f5j2z\" (UID: \"386123cf-4441-4cbf-962e-8c4c3e03dec1\") " pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.816301 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.816272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dh9\" (UniqueName: \"kubernetes.io/projected/386123cf-4441-4cbf-962e-8c4c3e03dec1-kube-api-access-m8dh9\") pod \"keda-admission-56f9798bb4-f5j2z\" (UID: \"386123cf-4441-4cbf-962e-8c4c3e03dec1\") " pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.895840 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.895753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:26.903733 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.903706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-certificates\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.906218 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.906197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9b706db2-ae11-4582-8cde-0862445ba072-certificates\") pod \"keda-metrics-apiserver-55557df8c-bkx86\" (UID: \"9b706db2-ae11-4582-8cde-0862445ba072\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:26.921618 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.921566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:26.935797 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.935519 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:26.996481 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:26.996306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:27.088672 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.088642 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-4fq6d"] Jul 15 10:50:27.090074 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:50:27.090045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73313f97_ee06_4f0e_8153_95cce511bc3a.slice/crio-d28a8e75f22b52d2cdc54f2d353281597e373fa73edb12a8a139d15501fecdc7 WatchSource:0}: Error finding container d28a8e75f22b52d2cdc54f2d353281597e373fa73edb12a8a139d15501fecdc7: Status 404 returned error can't find the container with id d28a8e75f22b52d2cdc54f2d353281597e373fa73edb12a8a139d15501fecdc7 Jul 15 10:50:27.117812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.117789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-f5j2z"] Jul 15 10:50:27.119871 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:50:27.119843 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386123cf_4441_4cbf_962e_8c4c3e03dec1.slice/crio-61a67a525f35a7e83d7d698c9e08bfd5402cf7c49f1f6c803239079712607e2e WatchSource:0}: Error finding container 61a67a525f35a7e83d7d698c9e08bfd5402cf7c49f1f6c803239079712607e2e: Status 404 returned error can't find the container with id 61a67a525f35a7e83d7d698c9e08bfd5402cf7c49f1f6c803239079712607e2e Jul 15 10:50:27.128770 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.128740 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:27.197663 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.197628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:27.269363 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.269150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-bkx86"] Jul 15 10:50:27.271548 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:50:27.271520 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b706db2_ae11_4582_8cde_0862445ba072.slice/crio-5aa4ac2728e936419048c4908e67f3fcc68f56355fc745cedf2d41dafdf85156 WatchSource:0}: Error finding container 5aa4ac2728e936419048c4908e67f3fcc68f56355fc745cedf2d41dafdf85156: Status 404 returned error can't find the container with id 5aa4ac2728e936419048c4908e67f3fcc68f56355fc745cedf2d41dafdf85156 Jul 15 10:50:27.357684 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.357641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:27.458675 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.458584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:27.970360 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.970292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" event={"ID":"386123cf-4441-4cbf-962e-8c4c3e03dec1","Type":"ContainerStarted","Data":"61a67a525f35a7e83d7d698c9e08bfd5402cf7c49f1f6c803239079712607e2e"} Jul 15 10:50:27.971781 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.971751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-4fq6d" event={"ID":"73313f97-ee06-4f0e-8153-95cce511bc3a","Type":"ContainerStarted","Data":"d28a8e75f22b52d2cdc54f2d353281597e373fa73edb12a8a139d15501fecdc7"} Jul 15 10:50:27.973070 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:27.973039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" event={"ID":"9b706db2-ae11-4582-8cde-0862445ba072","Type":"ContainerStarted","Data":"5aa4ac2728e936419048c4908e67f3fcc68f56355fc745cedf2d41dafdf85156"} Jul 15 10:50:30.990278 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:30.990215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" event={"ID":"9b706db2-ae11-4582-8cde-0862445ba072","Type":"ContainerStarted","Data":"7418beaf31b8c5a317f4f40f9072ff8defd1b6dbf9cc49f0239cfbce962c35c0"} Jul 15 10:50:30.990900 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:30.990846 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:30.992688 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:30.992661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-4fq6d" event={"ID":"73313f97-ee06-4f0e-8153-95cce511bc3a","Type":"ContainerStarted","Data":"86429eec41902bef898425290cf71e35d4539db05576b62accff6fdb800a0d69"} Jul 15 10:50:30.992976 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:30.992955 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:31.009028 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:31.008880 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" podStartSLOduration=1.416751721 podStartE2EDuration="5.008859361s" totalImagesPullingTime="3.59210764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:50:31.008365453 +0000 UTC m=+248.784816729" watchObservedRunningTime="2026-07-15 10:50:31.008859361 +0000 UTC m=+248.785310600" Jul 15 10:50:31.027546 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:31.027491 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-4fq6d" podStartSLOduration=2.248174913 podStartE2EDuration="6.027473665s" totalImagesPullingTime="3.779298752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:50:31.026347821 +0000 UTC m=+248.802799105" watchObservedRunningTime="2026-07-15 10:50:31.027473665 +0000 UTC m=+248.803924901" Jul 15 10:50:31.998092 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:31.997989 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" event={"ID":"386123cf-4441-4cbf-962e-8c4c3e03dec1","Type":"ContainerStarted","Data":"13dff0dbe861c30a82073951fb08453170ee8cbbedf58b2d3d90af49b2a49798"} Jul 15 10:50:31.998532 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:31.998175 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:50:32.014669 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:32.014620 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" podStartSLOduration=1.5084153649999998 podStartE2EDuration="6.014596103s" totalImagesPullingTime="4.506180738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:50:32.014194328 +0000 UTC m=+249.790645566" watchObservedRunningTime="2026-07-15 10:50:32.014596103 +0000 UTC m=+249.791047331" Jul 15 10:50:43.007850 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:43.007770 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bkx86" Jul 15 10:50:45.997554 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:45.997502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:46.052856 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:46.052815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:46.966938 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:46.966907 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-9vwbd" Jul 15 10:50:47.695536 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:47.695500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:47.783671 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:47.783634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:50:52.000713 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:52.000679 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-4fq6d" Jul 15 10:50:53.005849 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:50:53.005820 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-f5j2z" Jul 15 10:51:22.677357 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:22.677323 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:51:22.682507 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:22.682479 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:51:22.682849 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:22.682829 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 10:51:35.658726 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.658688 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-749567458d-k9t7c"] Jul 15 10:51:35.661182 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.661161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.663884 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.663866 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-stmgj\"" Jul 15 10:51:35.664769 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.664740 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 15 10:51:35.664900 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.664782 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 15 10:51:35.664900 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.664790 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 15 10:51:35.669370 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.669347 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-rtgm8"] Jul 15 10:51:35.671719 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.671700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:35.674084 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.674067 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-fhrhj\"" Jul 15 10:51:35.675596 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.675563 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 15 10:51:35.676510 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.676488 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-k9t7c"] Jul 15 10:51:35.681537 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.681505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:35.692747 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.692717 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-rtgm8"] Jul 15 10:51:35.710048 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.710008 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-5wbmv"] Jul 15 10:51:35.712396 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.712371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:35.714878 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.714857 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 15 10:51:35.714986 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.714952 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2rm8s\"" Jul 15 10:51:35.725995 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.725968 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5wbmv"] Jul 15 10:51:35.788395 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.788352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b49ebeb4-9383-48f3-8bc3-2882a1e30e02-data\") pod \"seaweedfs-64568bcd49-5wbmv\" (UID: \"b49ebeb4-9383-48f3-8bc3-2882a1e30e02\") " pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:35.788395 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.788395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqmv\" (UniqueName: \"kubernetes.io/projected/15a0648a-40fd-4481-b244-17dd729a3634-kube-api-access-6rqmv\") pod \"kserve-controller-manager-749567458d-k9t7c\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.788638 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.788474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjgn\" (UniqueName: \"kubernetes.io/projected/4c570fbb-952e-4fab-8dfe-4bf5acf78744-kube-api-access-xhjgn\") pod \"llmisvc-controller-manager-654c557856-rtgm8\" (UID: \"4c570fbb-952e-4fab-8dfe-4bf5acf78744\") " pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:35.788638 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.788505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a0648a-40fd-4481-b244-17dd729a3634-cert\") pod \"kserve-controller-manager-749567458d-k9t7c\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.788638 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.788531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzdp\" (UniqueName: \"kubernetes.io/projected/b49ebeb4-9383-48f3-8bc3-2882a1e30e02-kube-api-access-sjzdp\") pod \"seaweedfs-64568bcd49-5wbmv\" (UID: \"b49ebeb4-9383-48f3-8bc3-2882a1e30e02\") " pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:35.788737 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.788637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c570fbb-952e-4fab-8dfe-4bf5acf78744-cert\") pod \"llmisvc-controller-manager-654c557856-rtgm8\" (UID: \"4c570fbb-952e-4fab-8dfe-4bf5acf78744\") " pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:35.889638 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.889566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjgn\" (UniqueName: \"kubernetes.io/projected/4c570fbb-952e-4fab-8dfe-4bf5acf78744-kube-api-access-xhjgn\") pod \"llmisvc-controller-manager-654c557856-rtgm8\" (UID: \"4c570fbb-952e-4fab-8dfe-4bf5acf78744\") " pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:35.889638 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.889638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a0648a-40fd-4481-b244-17dd729a3634-cert\") pod \"kserve-controller-manager-749567458d-k9t7c\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.889904 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.889663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzdp\" (UniqueName: \"kubernetes.io/projected/b49ebeb4-9383-48f3-8bc3-2882a1e30e02-kube-api-access-sjzdp\") pod \"seaweedfs-64568bcd49-5wbmv\" (UID: \"b49ebeb4-9383-48f3-8bc3-2882a1e30e02\") " pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:35.889904 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.889712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c570fbb-952e-4fab-8dfe-4bf5acf78744-cert\") pod \"llmisvc-controller-manager-654c557856-rtgm8\" (UID: \"4c570fbb-952e-4fab-8dfe-4bf5acf78744\") " pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:35.889904 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.889731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b49ebeb4-9383-48f3-8bc3-2882a1e30e02-data\") pod \"seaweedfs-64568bcd49-5wbmv\" (UID: \"b49ebeb4-9383-48f3-8bc3-2882a1e30e02\") " pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:35.889904 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.889749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqmv\" (UniqueName: \"kubernetes.io/projected/15a0648a-40fd-4481-b244-17dd729a3634-kube-api-access-6rqmv\") pod \"kserve-controller-manager-749567458d-k9t7c\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.890156 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.890125 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b49ebeb4-9383-48f3-8bc3-2882a1e30e02-data\") pod \"seaweedfs-64568bcd49-5wbmv\" (UID: \"b49ebeb4-9383-48f3-8bc3-2882a1e30e02\") " pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:35.892172 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.892152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c570fbb-952e-4fab-8dfe-4bf5acf78744-cert\") pod \"llmisvc-controller-manager-654c557856-rtgm8\" (UID: \"4c570fbb-952e-4fab-8dfe-4bf5acf78744\") " pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:35.892248 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.892227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a0648a-40fd-4481-b244-17dd729a3634-cert\") pod \"kserve-controller-manager-749567458d-k9t7c\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.900033 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.900003 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqmv\" (UniqueName: \"kubernetes.io/projected/15a0648a-40fd-4481-b244-17dd729a3634-kube-api-access-6rqmv\") pod \"kserve-controller-manager-749567458d-k9t7c\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.900290 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.900266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzdp\" (UniqueName: \"kubernetes.io/projected/b49ebeb4-9383-48f3-8bc3-2882a1e30e02-kube-api-access-sjzdp\") pod \"seaweedfs-64568bcd49-5wbmv\" (UID: \"b49ebeb4-9383-48f3-8bc3-2882a1e30e02\") " pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:35.900431 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.900414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjgn\" (UniqueName: \"kubernetes.io/projected/4c570fbb-952e-4fab-8dfe-4bf5acf78744-kube-api-access-xhjgn\") pod \"llmisvc-controller-manager-654c557856-rtgm8\" (UID: \"4c570fbb-952e-4fab-8dfe-4bf5acf78744\") " pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:35.972711 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.972604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:35.982134 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:35.982105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:36.021900 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.021862 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:36.132041 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.132006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-k9t7c"] Jul 15 10:51:36.136067 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:51:36.136034 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a0648a_40fd_4481_b244_17dd729a3634.slice/crio-315ac3ae47d024eb9d84f9384df13c304afc36b062962d4a90114f7fcc6879a5 WatchSource:0}: Error finding container 315ac3ae47d024eb9d84f9384df13c304afc36b062962d4a90114f7fcc6879a5: Status 404 returned error can't find the container with id 315ac3ae47d024eb9d84f9384df13c304afc36b062962d4a90114f7fcc6879a5 Jul 15 10:51:36.137676 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.137658 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 10:51:36.152438 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.152404 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-rtgm8"] Jul 15 10:51:36.157063 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:51:36.157014 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c570fbb_952e_4fab_8dfe_4bf5acf78744.slice/crio-243dfed784b192cbb9841692a136b6e2b950dcad6100b3a847a509eaecec9c7d WatchSource:0}: Error finding container 243dfed784b192cbb9841692a136b6e2b950dcad6100b3a847a509eaecec9c7d: Status 404 returned error can't find the container with id 243dfed784b192cbb9841692a136b6e2b950dcad6100b3a847a509eaecec9c7d Jul 15 10:51:36.182795 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.182768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5wbmv"] Jul 15 10:51:36.185062 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:51:36.185035 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49ebeb4_9383_48f3_8bc3_2882a1e30e02.slice/crio-e405d9b8f3f505afcfc41c7070533ec123df06d5d6b480543777ba6f46892f88 WatchSource:0}: Error finding container e405d9b8f3f505afcfc41c7070533ec123df06d5d6b480543777ba6f46892f88: Status 404 returned error can't find the container with id e405d9b8f3f505afcfc41c7070533ec123df06d5d6b480543777ba6f46892f88 Jul 15 10:51:36.256478 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.256388 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5wbmv" event={"ID":"b49ebeb4-9383-48f3-8bc3-2882a1e30e02","Type":"ContainerStarted","Data":"e405d9b8f3f505afcfc41c7070533ec123df06d5d6b480543777ba6f46892f88"} Jul 15 10:51:36.257551 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.257522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" event={"ID":"4c570fbb-952e-4fab-8dfe-4bf5acf78744","Type":"ContainerStarted","Data":"243dfed784b192cbb9841692a136b6e2b950dcad6100b3a847a509eaecec9c7d"} Jul 15 10:51:36.264428 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:36.264397 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-k9t7c" event={"ID":"15a0648a-40fd-4481-b244-17dd729a3634","Type":"ContainerStarted","Data":"315ac3ae47d024eb9d84f9384df13c304afc36b062962d4a90114f7fcc6879a5"} Jul 15 10:51:37.157254 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:37.157217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:51:41.301315 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.301220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5wbmv" event={"ID":"b49ebeb4-9383-48f3-8bc3-2882a1e30e02","Type":"ContainerStarted","Data":"dc2379bc565d4aa1965e4ea7ad051f452887540b859aa9061345893745cbd522"} Jul 15 10:51:41.301315 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.301299 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:51:41.303925 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.303670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" event={"ID":"4c570fbb-952e-4fab-8dfe-4bf5acf78744","Type":"ContainerStarted","Data":"9ed097d6ad0e9cb006684c5f652a8b5ca7f8a46fe129c21e60863e6f1d0a8c6d"} Jul 15 10:51:41.305961 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.304141 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:51:41.306100 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.305974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-k9t7c" event={"ID":"15a0648a-40fd-4481-b244-17dd729a3634","Type":"ContainerStarted","Data":"5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5"} Jul 15 10:51:41.306620 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.306596 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:51:41.319889 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.319829 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-5wbmv" podStartSLOduration=1.442188994 podStartE2EDuration="6.319813087s" totalImagesPullingTime="4.877624093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:51:41.318207762 +0000 UTC m=+319.094659020" watchObservedRunningTime="2026-07-15 10:51:41.319813087 +0000 UTC m=+319.096264323" Jul 15 10:51:41.336273 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.336211 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" podStartSLOduration=1.483741205 podStartE2EDuration="6.336191925s" totalImagesPullingTime="4.85245072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:51:41.335652107 +0000 UTC m=+319.112103372" watchObservedRunningTime="2026-07-15 10:51:41.336191925 +0000 UTC m=+319.112643163" Jul 15 10:51:41.357057 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:41.356997 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-749567458d-k9t7c" podStartSLOduration=1.642302811 podStartE2EDuration="6.356977852s" totalImagesPullingTime="4.714675041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:51:41.355803347 +0000 UTC m=+319.132254598" watchObservedRunningTime="2026-07-15 10:51:41.356977852 +0000 UTC m=+319.133429090" Jul 15 10:51:47.313013 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:51:47.312982 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-5wbmv" Jul 15 10:52:12.312801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:12.312723 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-rtgm8" Jul 15 10:52:13.320399 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:13.320368 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:52:19.624034 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.623996 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-749567458d-k9t7c"] Jul 15 10:52:19.624475 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.624194 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-749567458d-k9t7c" podUID="15a0648a-40fd-4481-b244-17dd729a3634" containerName="manager" containerID="cri-o://5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5" gracePeriod=10 Jul 15 10:52:19.648639 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.648604 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-749567458d-kgznf"] Jul 15 10:52:19.650801 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.650779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:19.660480 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.660449 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-kgznf"] Jul 15 10:52:19.685101 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.685065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df683d0-3a9c-431a-9a9e-cbe1d94f0690-cert\") pod \"kserve-controller-manager-749567458d-kgznf\" (UID: \"4df683d0-3a9c-431a-9a9e-cbe1d94f0690\") " pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:19.685276 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.685121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hfj\" (UniqueName: \"kubernetes.io/projected/4df683d0-3a9c-431a-9a9e-cbe1d94f0690-kube-api-access-z9hfj\") pod \"kserve-controller-manager-749567458d-kgznf\" (UID: \"4df683d0-3a9c-431a-9a9e-cbe1d94f0690\") " pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:19.785641 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.785599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df683d0-3a9c-431a-9a9e-cbe1d94f0690-cert\") pod \"kserve-controller-manager-749567458d-kgznf\" (UID: \"4df683d0-3a9c-431a-9a9e-cbe1d94f0690\") " pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:19.785857 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.785699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hfj\" (UniqueName: \"kubernetes.io/projected/4df683d0-3a9c-431a-9a9e-cbe1d94f0690-kube-api-access-z9hfj\") pod \"kserve-controller-manager-749567458d-kgznf\" (UID: \"4df683d0-3a9c-431a-9a9e-cbe1d94f0690\") " pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:19.788199 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.788167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4df683d0-3a9c-431a-9a9e-cbe1d94f0690-cert\") pod \"kserve-controller-manager-749567458d-kgznf\" (UID: \"4df683d0-3a9c-431a-9a9e-cbe1d94f0690\") " pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:19.795773 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.795117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hfj\" (UniqueName: \"kubernetes.io/projected/4df683d0-3a9c-431a-9a9e-cbe1d94f0690-kube-api-access-z9hfj\") pod \"kserve-controller-manager-749567458d-kgznf\" (UID: \"4df683d0-3a9c-431a-9a9e-cbe1d94f0690\") " pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:19.872054 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.872023 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:52:19.987319 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.987224 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rqmv\" (UniqueName: \"kubernetes.io/projected/15a0648a-40fd-4481-b244-17dd729a3634-kube-api-access-6rqmv\") pod \"15a0648a-40fd-4481-b244-17dd729a3634\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " Jul 15 10:52:19.987319 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.987310 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a0648a-40fd-4481-b244-17dd729a3634-cert\") pod \"15a0648a-40fd-4481-b244-17dd729a3634\" (UID: \"15a0648a-40fd-4481-b244-17dd729a3634\") " Jul 15 10:52:19.989465 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.989437 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a0648a-40fd-4481-b244-17dd729a3634-cert" (OuterVolumeSpecName: "cert") pod "15a0648a-40fd-4481-b244-17dd729a3634" (UID: "15a0648a-40fd-4481-b244-17dd729a3634"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:52:19.989557 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:19.989485 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a0648a-40fd-4481-b244-17dd729a3634-kube-api-access-6rqmv" (OuterVolumeSpecName: "kube-api-access-6rqmv") pod "15a0648a-40fd-4481-b244-17dd729a3634" (UID: "15a0648a-40fd-4481-b244-17dd729a3634"). InnerVolumeSpecName "kube-api-access-6rqmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 10:52:20.005303 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.005271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:20.088313 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.088276 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqmv\" (UniqueName: \"kubernetes.io/projected/15a0648a-40fd-4481-b244-17dd729a3634-kube-api-access-6rqmv\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:52:20.088313 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.088305 2580 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15a0648a-40fd-4481-b244-17dd729a3634-cert\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:52:20.145816 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.145787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-kgznf"] Jul 15 10:52:20.148443 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:52:20.148414 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df683d0_3a9c_431a_9a9e_cbe1d94f0690.slice/crio-7666315e6ca3bf74783b96b06f9dce5cb1a62fd7086b17cf161e59fc4e6feb14 WatchSource:0}: Error finding container 7666315e6ca3bf74783b96b06f9dce5cb1a62fd7086b17cf161e59fc4e6feb14: Status 404 returned error can't find the container with id 7666315e6ca3bf74783b96b06f9dce5cb1a62fd7086b17cf161e59fc4e6feb14 Jul 15 10:52:20.478020 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.477980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-kgznf" event={"ID":"4df683d0-3a9c-431a-9a9e-cbe1d94f0690","Type":"ContainerStarted","Data":"7666315e6ca3bf74783b96b06f9dce5cb1a62fd7086b17cf161e59fc4e6feb14"} Jul 15 10:52:20.479258 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.479235 2580 generic.go:356] "Generic (PLEG): container finished" podID="15a0648a-40fd-4481-b244-17dd729a3634" containerID="5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5" exitCode=0 Jul 15 10:52:20.479343 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.479286 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-k9t7c" Jul 15 10:52:20.479343 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.479323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-k9t7c" event={"ID":"15a0648a-40fd-4481-b244-17dd729a3634","Type":"ContainerDied","Data":"5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5"} Jul 15 10:52:20.479433 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.479376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-k9t7c" event={"ID":"15a0648a-40fd-4481-b244-17dd729a3634","Type":"ContainerDied","Data":"315ac3ae47d024eb9d84f9384df13c304afc36b062962d4a90114f7fcc6879a5"} Jul 15 10:52:20.479433 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.479416 2580 scope.go:117] "RemoveContainer" containerID="5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5" Jul 15 10:52:20.488478 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.488458 2580 scope.go:117] "RemoveContainer" containerID="5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5" Jul 15 10:52:20.488827 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:52:20.488805 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5\": container with ID starting with 5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5 not found: ID does not exist" containerID="5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5" Jul 15 10:52:20.488906 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.488834 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5"} err="failed to get container status \"5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5\": rpc error: code = NotFound desc = could not find container \"5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5\": container with ID starting with 5c5f113f785b000655ff07a8668be609858170e543cabdb1b5bb1bf7a05670e5 not found: ID does not exist" Jul 15 10:52:20.502376 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.502332 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-749567458d-k9t7c"] Jul 15 10:52:20.503954 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.503933 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-749567458d-k9t7c"] Jul 15 10:52:20.826981 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:20.826946 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a0648a-40fd-4481-b244-17dd729a3634" path="/var/lib/kubelet/pods/15a0648a-40fd-4481-b244-17dd729a3634/volumes" Jul 15 10:52:21.485506 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:21.485469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-kgznf" event={"ID":"4df683d0-3a9c-431a-9a9e-cbe1d94f0690","Type":"ContainerStarted","Data":"d4b4af28b1074fee2d5c1ea370df9fb69ef6345e48f8c11f95d4011af796d65c"} Jul 15 10:52:21.485743 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:21.485709 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:21.505760 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:21.505705 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-749567458d-kgznf" podStartSLOduration=1.9202114799999999 podStartE2EDuration="2.505686366s" totalImagesPullingTime="585.474886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:52:21.503215943 +0000 UTC m=+359.279667180" watchObservedRunningTime="2026-07-15 10:52:21.505686366 +0000 UTC m=+359.282137603" Jul 15 10:52:39.069537 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:39.069500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:39.164391 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:39.164357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:41.012384 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:41.012349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:41.070385 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:41.070351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:52.495102 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:52.495068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-749567458d-kgznf" Jul 15 10:52:53.477540 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.477504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:53.516562 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.516529 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-55kc6"] Jul 15 10:52:53.516955 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.516909 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a0648a-40fd-4481-b244-17dd729a3634" containerName="manager" Jul 15 10:52:53.516955 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.516921 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a0648a-40fd-4481-b244-17dd729a3634" containerName="manager" Jul 15 10:52:53.517023 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.516975 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a0648a-40fd-4481-b244-17dd729a3634" containerName="manager" Jul 15 10:52:53.520207 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.520188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.522760 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.522735 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 15 10:52:53.522890 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.522870 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-bctct\"" Jul 15 10:52:53.531508 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.531477 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-55kc6"] Jul 15 10:52:53.534817 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.534793 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xppm4"] Jul 15 10:52:53.537988 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.537970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:53.540176 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.540146 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 15 10:52:53.540313 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.540236 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wwz2k\"" Jul 15 10:52:53.548171 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.548144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xppm4"] Jul 15 10:52:53.568734 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.568698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:53.586878 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.586848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-cert\") pod \"odh-model-controller-84b54b7f97-xppm4\" (UID: \"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d\") " pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:53.586986 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.586903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx2qv\" (UniqueName: \"kubernetes.io/projected/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-kube-api-access-bx2qv\") pod \"odh-model-controller-84b54b7f97-xppm4\" (UID: \"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d\") " pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:53.688136 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.688096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2mv\" (UniqueName: \"kubernetes.io/projected/a8788539-8dca-4714-a3c1-48d9858ad9a0-kube-api-access-5c2mv\") pod \"model-serving-api-fd65d7d6-55kc6\" (UID: \"a8788539-8dca-4714-a3c1-48d9858ad9a0\") " pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.688343 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.688147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8788539-8dca-4714-a3c1-48d9858ad9a0-tls-certs\") pod \"model-serving-api-fd65d7d6-55kc6\" (UID: \"a8788539-8dca-4714-a3c1-48d9858ad9a0\") " pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.688343 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.688228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-cert\") pod \"odh-model-controller-84b54b7f97-xppm4\" (UID: \"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d\") " pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:53.688343 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.688291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx2qv\" (UniqueName: \"kubernetes.io/projected/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-kube-api-access-bx2qv\") pod \"odh-model-controller-84b54b7f97-xppm4\" (UID: \"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d\") " pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:53.688504 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:52:53.688396 2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 10:52:53.688504 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:52:53.688458 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-cert podName:8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d nodeName:}" failed. No retries permitted until 2026-07-15 10:52:54.188439266 +0000 UTC m=+391.964890481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-cert") pod "odh-model-controller-84b54b7f97-xppm4" (UID: "8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d") : secret "odh-model-controller-webhook-cert" not found Jul 15 10:52:53.695863 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.695825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:53.697755 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.697722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx2qv\" (UniqueName: \"kubernetes.io/projected/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-kube-api-access-bx2qv\") pod \"odh-model-controller-84b54b7f97-xppm4\" (UID: \"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d\") " pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:53.789651 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.789531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2mv\" (UniqueName: \"kubernetes.io/projected/a8788539-8dca-4714-a3c1-48d9858ad9a0-kube-api-access-5c2mv\") pod \"model-serving-api-fd65d7d6-55kc6\" (UID: \"a8788539-8dca-4714-a3c1-48d9858ad9a0\") " pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.789651 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.789603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8788539-8dca-4714-a3c1-48d9858ad9a0-tls-certs\") pod \"model-serving-api-fd65d7d6-55kc6\" (UID: \"a8788539-8dca-4714-a3c1-48d9858ad9a0\") " pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.792264 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.792234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8788539-8dca-4714-a3c1-48d9858ad9a0-tls-certs\") pod \"model-serving-api-fd65d7d6-55kc6\" (UID: \"a8788539-8dca-4714-a3c1-48d9858ad9a0\") " pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.796273 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.796223 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:53.799800 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.799769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2mv\" (UniqueName: \"kubernetes.io/projected/a8788539-8dca-4714-a3c1-48d9858ad9a0-kube-api-access-5c2mv\") pod \"model-serving-api-fd65d7d6-55kc6\" (UID: \"a8788539-8dca-4714-a3c1-48d9858ad9a0\") " pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.831679 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.831644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:53.968821 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:53.968788 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-55kc6"] Jul 15 10:52:53.973059 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:52:53.973027 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8788539_8dca_4714_a3c1_48d9858ad9a0.slice/crio-3c3e7eece14ad11c5c5677cc63119e098fb29725cc4d66fd4960a0093077b728 WatchSource:0}: Error finding container 3c3e7eece14ad11c5c5677cc63119e098fb29725cc4d66fd4960a0093077b728: Status 404 returned error can't find the container with id 3c3e7eece14ad11c5c5677cc63119e098fb29725cc4d66fd4960a0093077b728 Jul 15 10:52:54.139619 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.139560 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:54.194621 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.194562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-cert\") pod \"odh-model-controller-84b54b7f97-xppm4\" (UID: \"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d\") " pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:54.197033 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.197008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d-cert\") pod \"odh-model-controller-84b54b7f97-xppm4\" (UID: \"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d\") " pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:54.240363 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.240333 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:52:54.450608 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.450497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:54.609754 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.609696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xppm4"] Jul 15 10:52:54.612673 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:52:54.612630 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1162d4_4b2c_4fb6_a43f_658e2a9baa5d.slice/crio-46ee32336d1856a02593133fc65f66b4a4e46f2afe2be7c31f154f7b62cbecb7 WatchSource:0}: Error finding container 46ee32336d1856a02593133fc65f66b4a4e46f2afe2be7c31f154f7b62cbecb7: Status 404 returned error can't find the container with id 46ee32336d1856a02593133fc65f66b4a4e46f2afe2be7c31f154f7b62cbecb7 Jul 15 10:52:54.631418 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.631342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-55kc6" event={"ID":"a8788539-8dca-4714-a3c1-48d9858ad9a0","Type":"ContainerStarted","Data":"3c3e7eece14ad11c5c5677cc63119e098fb29725cc4d66fd4960a0093077b728"} Jul 15 10:52:54.633041 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:54.633008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xppm4" event={"ID":"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d","Type":"ContainerStarted","Data":"46ee32336d1856a02593133fc65f66b4a4e46f2afe2be7c31f154f7b62cbecb7"} Jul 15 10:52:56.647535 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:56.647403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-55kc6" event={"ID":"a8788539-8dca-4714-a3c1-48d9858ad9a0","Type":"ContainerStarted","Data":"80ac1b1d35a7b837bb2ff7037f88bcf48d0d1b6e5343883125a219d2429a732d"} Jul 15 10:52:56.648002 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:56.647535 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:52:56.669879 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:56.669806 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-55kc6" podStartSLOduration=1.2761070939999999 podStartE2EDuration="3.669785228s" totalImagesPullingTime="2.393678134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:52:56.667770937 +0000 UTC m=+394.444222205" watchObservedRunningTime="2026-07-15 10:52:56.669785228 +0000 UTC m=+394.446236508" Jul 15 10:52:59.664003 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:59.663963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xppm4" event={"ID":"8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d","Type":"ContainerStarted","Data":"739033d22097cddb4c8784525ed53d24c2b9367b62659bf6a173620522087062"} Jul 15 10:52:59.664362 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:59.664067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:52:59.682071 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:52:59.682023 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-xppm4" podStartSLOduration=2.658898121 podStartE2EDuration="6.682004555s" totalImagesPullingTime="4.023106434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:52:59.681097968 +0000 UTC m=+397.457549224" watchObservedRunningTime="2026-07-15 10:52:59.682004555 +0000 UTC m=+397.458455791" Jul 15 10:53:00.149175 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:00.149140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:00.250743 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:00.250715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:07.657379 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:07.657349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-55kc6" Jul 15 10:53:10.671954 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:10.671918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-xppm4" Jul 15 10:53:11.559589 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:11.559535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:11.613944 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:11.613909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:11.675835 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:11.675789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:53:55.355033 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:53:55.354948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:05.245506 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:05.245473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:05.310800 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:05.310765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:30.073391 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:30.073354 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:30.177312 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:30.177277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:42.667326 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:42.667278 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:42.987860 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:42.987768 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:43.103503 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:43.103455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:43.278476 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:43.278389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:43.362282 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:43.362247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:43.478056 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:43.478023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:43.651169 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:43.651137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:43.754868 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:43.754836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:43.837606 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:43.837558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:44.170194 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:44.170163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:46.832903 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:46.832870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:54:46.900823 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:54:46.900785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:34.752977 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:55:34.752876 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:34.966443 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:55:34.966409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:55:45.634904 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:55:45.634871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:04.288383 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:04.288295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:22.709200 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.709173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:56:22.709782 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.709649 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 10:56:22.763561 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.763530 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t"] Jul 15 10:56:22.765958 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.765939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:22.768358 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.768333 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6db9e-serving-cert\"" Jul 15 10:56:22.768358 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.768333 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6db9e-kube-rbac-proxy-sar-config\"" Jul 15 10:56:22.768759 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.768742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nfb6q\"" Jul 15 10:56:22.769293 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.769268 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 15 10:56:22.770055 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.770035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:22.778849 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.778817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t"] Jul 15 10:56:22.791420 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.791379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls\") pod \"switch-graph-6db9e-5cb89b98dc-7x78t\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:22.791655 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.791512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3760677f-62e7-4d20-bb5a-a8dca199de7d-openshift-service-ca-bundle\") pod \"switch-graph-6db9e-5cb89b98dc-7x78t\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:22.892311 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.892275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3760677f-62e7-4d20-bb5a-a8dca199de7d-openshift-service-ca-bundle\") pod \"switch-graph-6db9e-5cb89b98dc-7x78t\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:22.892486 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.892324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls\") pod \"switch-graph-6db9e-5cb89b98dc-7x78t\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:22.892486 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:56:22.892421 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-6db9e-serving-cert: secret "switch-graph-6db9e-serving-cert" not found Jul 15 10:56:22.892558 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:56:22.892488 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls podName:3760677f-62e7-4d20-bb5a-a8dca199de7d nodeName:}" failed. No retries permitted until 2026-07-15 10:56:23.392471586 +0000 UTC m=+601.168922804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls") pod "switch-graph-6db9e-5cb89b98dc-7x78t" (UID: "3760677f-62e7-4d20-bb5a-a8dca199de7d") : secret "switch-graph-6db9e-serving-cert" not found Jul 15 10:56:22.893004 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.892983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3760677f-62e7-4d20-bb5a-a8dca199de7d-openshift-service-ca-bundle\") pod \"switch-graph-6db9e-5cb89b98dc-7x78t\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:22.969785 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:22.969687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:23.397386 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:23.397342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls\") pod \"switch-graph-6db9e-5cb89b98dc-7x78t\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:23.399812 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:23.399787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls\") pod \"switch-graph-6db9e-5cb89b98dc-7x78t\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:23.677005 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:23.676913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:23.814519 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:23.814493 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t"] Jul 15 10:56:23.817202 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:56:23.817168 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3760677f_62e7_4d20_bb5a_a8dca199de7d.slice/crio-9d5b7d7965ed6fe83d14263ef11d77840ca1fb08a65b21488c19aeb434cd6a52 WatchSource:0}: Error finding container 9d5b7d7965ed6fe83d14263ef11d77840ca1fb08a65b21488c19aeb434cd6a52: Status 404 returned error can't find the container with id 9d5b7d7965ed6fe83d14263ef11d77840ca1fb08a65b21488c19aeb434cd6a52 Jul 15 10:56:24.607939 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:24.607902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" event={"ID":"3760677f-62e7-4d20-bb5a-a8dca199de7d","Type":"ContainerStarted","Data":"9d5b7d7965ed6fe83d14263ef11d77840ca1fb08a65b21488c19aeb434cd6a52"} Jul 15 10:56:26.619364 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:26.619329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" event={"ID":"3760677f-62e7-4d20-bb5a-a8dca199de7d","Type":"ContainerStarted","Data":"d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327"} Jul 15 10:56:26.619796 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:26.619431 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:26.641122 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:26.641070 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podStartSLOduration=2.402095009 podStartE2EDuration="4.641054873s" totalImagesPullingTime="2.238959864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:26.639133081 +0000 UTC m=+604.415584346" watchObservedRunningTime="2026-07-15 10:56:26.641054873 +0000 UTC m=+604.417506109" Jul 15 10:56:32.630630 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:32.630601 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:32.949156 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:32.949060 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t"] Jul 15 10:56:32.949317 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:32.949252 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" containerID="cri-o://d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327" gracePeriod=30 Jul 15 10:56:32.971591 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:32.971550 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:33.143824 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:33.143784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:33.252338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:33.252254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:33.325440 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:33.325409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:33.389585 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:33.389545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:56:37.628669 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:37.628626 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:56:42.628344 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:42.628252 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:56:47.629185 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:47.629136 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:56:47.629622 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:47.629246 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:56:52.628777 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:52.628732 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:56:57.628103 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:56:57.628056 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:02.629082 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.629038 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:02.812596 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.812550 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf"] Jul 15 10:57:02.816161 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.816133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:02.817221 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.817192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:02.818846 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.818825 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 15 10:57:02.818953 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.818877 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 15 10:57:02.827774 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.827751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf"] Jul 15 10:57:02.956163 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.956140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b09147f-02ce-48c0-a197-d696e1ab1946-openshift-service-ca-bundle\") pod \"model-chainer-7fdbd8cf64-8vlgf\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:02.956227 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:02.956177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b09147f-02ce-48c0-a197-d696e1ab1946-proxy-tls\") pod \"model-chainer-7fdbd8cf64-8vlgf\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:03.056977 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.056939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b09147f-02ce-48c0-a197-d696e1ab1946-openshift-service-ca-bundle\") pod \"model-chainer-7fdbd8cf64-8vlgf\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:03.057144 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.056989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b09147f-02ce-48c0-a197-d696e1ab1946-proxy-tls\") pod \"model-chainer-7fdbd8cf64-8vlgf\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:03.057689 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.057663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b09147f-02ce-48c0-a197-d696e1ab1946-openshift-service-ca-bundle\") pod \"model-chainer-7fdbd8cf64-8vlgf\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:03.059585 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.059540 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b09147f-02ce-48c0-a197-d696e1ab1946-proxy-tls\") pod \"model-chainer-7fdbd8cf64-8vlgf\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:03.107906 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.107879 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:57:03.129137 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.129107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:03.258819 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.258779 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls\") pod \"3760677f-62e7-4d20-bb5a-a8dca199de7d\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " Jul 15 10:57:03.259002 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.258876 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3760677f-62e7-4d20-bb5a-a8dca199de7d-openshift-service-ca-bundle\") pod \"3760677f-62e7-4d20-bb5a-a8dca199de7d\" (UID: \"3760677f-62e7-4d20-bb5a-a8dca199de7d\") " Jul 15 10:57:03.259234 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.259211 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3760677f-62e7-4d20-bb5a-a8dca199de7d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3760677f-62e7-4d20-bb5a-a8dca199de7d" (UID: "3760677f-62e7-4d20-bb5a-a8dca199de7d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:57:03.261015 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.260986 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3760677f-62e7-4d20-bb5a-a8dca199de7d" (UID: "3760677f-62e7-4d20-bb5a-a8dca199de7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:57:03.265472 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.265448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf"] Jul 15 10:57:03.267615 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:57:03.267586 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b09147f_02ce_48c0_a197_d696e1ab1946.slice/crio-d4256ad3593463fa2e547140f628a2b32789e0a92bcb2af5cc358938f8644ba6 WatchSource:0}: Error finding container d4256ad3593463fa2e547140f628a2b32789e0a92bcb2af5cc358938f8644ba6: Status 404 returned error can't find the container with id d4256ad3593463fa2e547140f628a2b32789e0a92bcb2af5cc358938f8644ba6 Jul 15 10:57:03.360087 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.360046 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3760677f-62e7-4d20-bb5a-a8dca199de7d-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:57:03.360087 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.360081 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3760677f-62e7-4d20-bb5a-a8dca199de7d-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:57:03.790382 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.790348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" event={"ID":"2b09147f-02ce-48c0-a197-d696e1ab1946","Type":"ContainerStarted","Data":"79198c81d0ad6e65f949ecc9c6ac1bee35abf20a6471030e41786018b375eada"} Jul 15 10:57:03.790875 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.790408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" event={"ID":"2b09147f-02ce-48c0-a197-d696e1ab1946","Type":"ContainerStarted","Data":"d4256ad3593463fa2e547140f628a2b32789e0a92bcb2af5cc358938f8644ba6"} Jul 15 10:57:03.790875 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.790440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:03.791489 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.791463 2580 generic.go:356] "Generic (PLEG): container finished" podID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerID="d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327" exitCode=0 Jul 15 10:57:03.791592 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.791503 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" Jul 15 10:57:03.791592 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.791532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" event={"ID":"3760677f-62e7-4d20-bb5a-a8dca199de7d","Type":"ContainerDied","Data":"d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327"} Jul 15 10:57:03.791670 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.791595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t" event={"ID":"3760677f-62e7-4d20-bb5a-a8dca199de7d","Type":"ContainerDied","Data":"9d5b7d7965ed6fe83d14263ef11d77840ca1fb08a65b21488c19aeb434cd6a52"} Jul 15 10:57:03.791670 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.791628 2580 scope.go:117] "RemoveContainer" containerID="d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327" Jul 15 10:57:03.800931 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.800912 2580 scope.go:117] "RemoveContainer" containerID="d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327" Jul 15 10:57:03.801230 ip-10-0-130-140 kubenswrapper[2580]: E0715 10:57:03.801211 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327\": container with ID starting with d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327 not found: ID does not exist" containerID="d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327" Jul 15 10:57:03.801287 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.801238 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327"} err="failed to get container status \"d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327\": rpc error: code = NotFound desc = could not find container \"d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327\": container with ID starting with d3905241c6d9d87abfdf40fddc3a278838ef975144573ddd6d15766176da9327 not found: ID does not exist" Jul 15 10:57:03.807546 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.807489 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podStartSLOduration=1.8074756010000002 podStartE2EDuration="1.807475601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:57:03.806967098 +0000 UTC m=+641.583418336" watchObservedRunningTime="2026-07-15 10:57:03.807475601 +0000 UTC m=+641.583926839" Jul 15 10:57:03.819828 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.819797 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t"] Jul 15 10:57:03.823658 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:03.823631 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t"] Jul 15 10:57:04.230244 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:04.230208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:04.825487 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:04.825449 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" path="/var/lib/kubelet/pods/3760677f-62e7-4d20-bb5a-a8dca199de7d/volumes" Jul 15 10:57:09.804458 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:09.804425 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:12.900471 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:12.900434 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf"] Jul 15 10:57:12.900956 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:12.900745 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" containerID="cri-o://79198c81d0ad6e65f949ecc9c6ac1bee35abf20a6471030e41786018b375eada" gracePeriod=30 Jul 15 10:57:12.922858 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:12.922827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:13.041825 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:13.041779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:13.138992 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:13.138951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:13.205640 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:13.205539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:13.302722 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:13.302692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:13.403556 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:13.403525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:14.803268 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:14.803230 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:19.802867 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:19.802827 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:24.803741 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:24.803703 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:24.804141 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:24.803832 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:29.802452 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:29.802413 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:33.241170 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.241130 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572"] Jul 15 10:57:33.241727 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.241689 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" Jul 15 10:57:33.241727 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.241711 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" Jul 15 10:57:33.241927 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.241804 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3760677f-62e7-4d20-bb5a-a8dca199de7d" containerName="switch-graph-6db9e" Jul 15 10:57:33.242566 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.242541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:33.246247 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.246226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.249150 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.249124 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ff4f8-serving-cert\"" Jul 15 10:57:33.249330 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.249317 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ff4f8-kube-rbac-proxy-sar-config\"" Jul 15 10:57:33.256493 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.256469 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572"] Jul 15 10:57:33.323713 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.323683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3e32df-59f4-4a0e-86b6-4e707c70d606-proxy-tls\") pod \"switch-graph-ff4f8-fc4884fc4-c9572\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.323885 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.323823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd3e32df-59f4-4a0e-86b6-4e707c70d606-openshift-service-ca-bundle\") pod \"switch-graph-ff4f8-fc4884fc4-c9572\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.360694 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.360658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:57:33.425066 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.425023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd3e32df-59f4-4a0e-86b6-4e707c70d606-openshift-service-ca-bundle\") pod \"switch-graph-ff4f8-fc4884fc4-c9572\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.425066 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.425065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3e32df-59f4-4a0e-86b6-4e707c70d606-proxy-tls\") pod \"switch-graph-ff4f8-fc4884fc4-c9572\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.425764 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.425745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd3e32df-59f4-4a0e-86b6-4e707c70d606-openshift-service-ca-bundle\") pod \"switch-graph-ff4f8-fc4884fc4-c9572\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.427560 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.427542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3e32df-59f4-4a0e-86b6-4e707c70d606-proxy-tls\") pod \"switch-graph-ff4f8-fc4884fc4-c9572\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.556884 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.556795 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.695362 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.695335 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572"] Jul 15 10:57:33.697684 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:57:33.697651 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3e32df_59f4_4a0e_86b6_4e707c70d606.slice/crio-09b8c51f762a5875c9ea07da3330e25c6ebe8420a17f8e63dc013a6367b70f29 WatchSource:0}: Error finding container 09b8c51f762a5875c9ea07da3330e25c6ebe8420a17f8e63dc013a6367b70f29: Status 404 returned error can't find the container with id 09b8c51f762a5875c9ea07da3330e25c6ebe8420a17f8e63dc013a6367b70f29 Jul 15 10:57:33.940669 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.940614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" event={"ID":"cd3e32df-59f4-4a0e-86b6-4e707c70d606","Type":"ContainerStarted","Data":"d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5"} Jul 15 10:57:33.940875 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.940690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" event={"ID":"cd3e32df-59f4-4a0e-86b6-4e707c70d606","Type":"ContainerStarted","Data":"09b8c51f762a5875c9ea07da3330e25c6ebe8420a17f8e63dc013a6367b70f29"} Jul 15 10:57:33.940875 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.940817 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:33.957985 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:33.957934 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podStartSLOduration=0.957916345 podStartE2EDuration="957.916345ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:57:33.956991233 +0000 UTC m=+671.733442482" watchObservedRunningTime="2026-07-15 10:57:33.957916345 +0000 UTC m=+671.734367960" Jul 15 10:57:34.802802 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:34.802762 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:39.802517 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:39.802471 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 10:57:39.951292 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:39.951255 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 10:57:42.986895 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:42.986860 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerID="79198c81d0ad6e65f949ecc9c6ac1bee35abf20a6471030e41786018b375eada" exitCode=0 Jul 15 10:57:42.987321 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:42.986939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" event={"ID":"2b09147f-02ce-48c0-a197-d696e1ab1946","Type":"ContainerDied","Data":"79198c81d0ad6e65f949ecc9c6ac1bee35abf20a6471030e41786018b375eada"} Jul 15 10:57:43.061012 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.060985 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:43.212275 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.212182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b09147f-02ce-48c0-a197-d696e1ab1946-openshift-service-ca-bundle\") pod \"2b09147f-02ce-48c0-a197-d696e1ab1946\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " Jul 15 10:57:43.212275 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.212255 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b09147f-02ce-48c0-a197-d696e1ab1946-proxy-tls\") pod \"2b09147f-02ce-48c0-a197-d696e1ab1946\" (UID: \"2b09147f-02ce-48c0-a197-d696e1ab1946\") " Jul 15 10:57:43.212566 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.212542 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b09147f-02ce-48c0-a197-d696e1ab1946-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2b09147f-02ce-48c0-a197-d696e1ab1946" (UID: "2b09147f-02ce-48c0-a197-d696e1ab1946"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 10:57:43.214401 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.214368 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b09147f-02ce-48c0-a197-d696e1ab1946-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b09147f-02ce-48c0-a197-d696e1ab1946" (UID: "2b09147f-02ce-48c0-a197-d696e1ab1946"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 10:57:43.313910 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.313869 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b09147f-02ce-48c0-a197-d696e1ab1946-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:57:43.313910 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.313900 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b09147f-02ce-48c0-a197-d696e1ab1946-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 10:57:43.993383 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.993350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" event={"ID":"2b09147f-02ce-48c0-a197-d696e1ab1946","Type":"ContainerDied","Data":"d4256ad3593463fa2e547140f628a2b32789e0a92bcb2af5cc358938f8644ba6"} Jul 15 10:57:43.993383 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.993370 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf" Jul 15 10:57:43.993920 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:43.993413 2580 scope.go:117] "RemoveContainer" containerID="79198c81d0ad6e65f949ecc9c6ac1bee35abf20a6471030e41786018b375eada" Jul 15 10:57:44.014332 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:44.014299 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf"] Jul 15 10:57:44.018836 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:44.018803 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf"] Jul 15 10:57:44.826338 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:57:44.826302 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" path="/var/lib/kubelet/pods/2b09147f-02ce-48c0-a197-d696e1ab1946/volumes" Jul 15 10:58:13.136794 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.136697 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:13.141793 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.141762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78"] Jul 15 10:58:13.142194 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.142176 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" Jul 15 10:58:13.142269 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.142196 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" Jul 15 10:58:13.142324 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.142273 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b09147f-02ce-48c0-a197-d696e1ab1946" containerName="model-chainer" Jul 15 10:58:13.145358 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.145336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.147840 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.147818 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-074d2-serving-cert\"" Jul 15 10:58:13.147978 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.147845 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-074d2-kube-rbac-proxy-sar-config\"" Jul 15 10:58:13.159268 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.159236 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78"] Jul 15 10:58:13.281360 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.281318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bfd496a-bb37-4b4a-bad2-c567178570e3-openshift-service-ca-bundle\") pod \"sequence-graph-074d2-574bc49b98-k4s78\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.281535 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.281374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bfd496a-bb37-4b4a-bad2-c567178570e3-proxy-tls\") pod \"sequence-graph-074d2-574bc49b98-k4s78\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.362901 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.362849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 10:58:13.382745 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.382712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bfd496a-bb37-4b4a-bad2-c567178570e3-openshift-service-ca-bundle\") pod \"sequence-graph-074d2-574bc49b98-k4s78\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.382852 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.382761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bfd496a-bb37-4b4a-bad2-c567178570e3-proxy-tls\") pod \"sequence-graph-074d2-574bc49b98-k4s78\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.383406 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.383375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bfd496a-bb37-4b4a-bad2-c567178570e3-openshift-service-ca-bundle\") pod \"sequence-graph-074d2-574bc49b98-k4s78\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.385211 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.385190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bfd496a-bb37-4b4a-bad2-c567178570e3-proxy-tls\") pod \"sequence-graph-074d2-574bc49b98-k4s78\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.455558 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.455461 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:13.593687 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:13.593657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78"] Jul 15 10:58:13.597736 ip-10-0-130-140 kubenswrapper[2580]: W0715 10:58:13.597706 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfd496a_bb37_4b4a_bad2_c567178570e3.slice/crio-a8376c46b49112c72f2fb0a62eb3b979e2c7a89163d3e5029277631afaa1143a WatchSource:0}: Error finding container a8376c46b49112c72f2fb0a62eb3b979e2c7a89163d3e5029277631afaa1143a: Status 404 returned error can't find the container with id a8376c46b49112c72f2fb0a62eb3b979e2c7a89163d3e5029277631afaa1143a Jul 15 10:58:14.145077 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:14.145041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" event={"ID":"3bfd496a-bb37-4b4a-bad2-c567178570e3","Type":"ContainerStarted","Data":"a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843"} Jul 15 10:58:14.145474 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:14.145090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" event={"ID":"3bfd496a-bb37-4b4a-bad2-c567178570e3","Type":"ContainerStarted","Data":"a8376c46b49112c72f2fb0a62eb3b979e2c7a89163d3e5029277631afaa1143a"} Jul 15 10:58:14.145474 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:14.145122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 10:58:14.163639 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:14.163590 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podStartSLOduration=1.163557559 podStartE2EDuration="1.163557559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:58:14.162560536 +0000 UTC m=+711.939011801" watchObservedRunningTime="2026-07-15 10:58:14.163557559 +0000 UTC m=+711.940008795" Jul 15 10:58:20.155798 ip-10-0-130-140 kubenswrapper[2580]: I0715 10:58:20.155763 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 11:00:38.947303 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:00:38.947261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:01:22.738831 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:01:22.738750 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:01:22.741082 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:01:22.741060 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:03:09.900013 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:03:09.899975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:47.755096 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:47.755001 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572"] Jul 15 11:05:47.755662 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:47.755262 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" containerID="cri-o://d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5" gracePeriod=30 Jul 15 11:05:47.831023 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:47.830982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:47.927122 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:47.927042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:48.036633 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:48.036528 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:48.137771 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:48.137726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:48.239583 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:48.239533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:05:49.949840 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:49.949795 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:05:54.949366 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:54.949327 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:05:59.949502 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:59.949465 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:05:59.949914 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:05:59.949586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 11:06:04.949973 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:04.949929 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:09.950012 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:09.949967 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:14.949703 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:14.949651 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:17.907557 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:17.907530 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 11:06:18.003287 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.003248 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd3e32df-59f4-4a0e-86b6-4e707c70d606-openshift-service-ca-bundle\") pod \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " Jul 15 11:06:18.003658 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.003631 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3e32df-59f4-4a0e-86b6-4e707c70d606-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cd3e32df-59f4-4a0e-86b6-4e707c70d606" (UID: "cd3e32df-59f4-4a0e-86b6-4e707c70d606"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:06:18.104150 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.104116 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3e32df-59f4-4a0e-86b6-4e707c70d606-proxy-tls\") pod \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\" (UID: \"cd3e32df-59f4-4a0e-86b6-4e707c70d606\") " Jul 15 11:06:18.104313 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.104290 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd3e32df-59f4-4a0e-86b6-4e707c70d606-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:06:18.106084 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.106054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3e32df-59f4-4a0e-86b6-4e707c70d606-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd3e32df-59f4-4a0e-86b6-4e707c70d606" (UID: "cd3e32df-59f4-4a0e-86b6-4e707c70d606"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:06:18.204841 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.204790 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3e32df-59f4-4a0e-86b6-4e707c70d606-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:06:18.358064 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.357972 2580 generic.go:356] "Generic (PLEG): container finished" podID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerID="d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5" exitCode=0 Jul 15 11:06:18.358064 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.358007 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" Jul 15 11:06:18.358064 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.358041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" event={"ID":"cd3e32df-59f4-4a0e-86b6-4e707c70d606","Type":"ContainerDied","Data":"d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5"} Jul 15 11:06:18.358336 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.358082 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572" event={"ID":"cd3e32df-59f4-4a0e-86b6-4e707c70d606","Type":"ContainerDied","Data":"09b8c51f762a5875c9ea07da3330e25c6ebe8420a17f8e63dc013a6367b70f29"} Jul 15 11:06:18.358336 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.358102 2580 scope.go:117] "RemoveContainer" containerID="d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5" Jul 15 11:06:18.367232 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.367214 2580 scope.go:117] "RemoveContainer" containerID="d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5" Jul 15 11:06:18.367553 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:06:18.367534 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5\": container with ID starting with d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5 not found: ID does not exist" containerID="d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5" Jul 15 11:06:18.367628 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.367564 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5"} err="failed to get container status \"d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5\": rpc error: code = NotFound desc = could not find container \"d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5\": container with ID starting with d2973198801221fcf9f4bb245d6da8553604664f993d0453eda1a74920133bf5 not found: ID does not exist" Jul 15 11:06:18.383317 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.383276 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572"] Jul 15 11:06:18.388538 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.388507 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572"] Jul 15 11:06:18.825352 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:18.825267 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" path="/var/lib/kubelet/pods/cd3e32df-59f4-4a0e-86b6-4e707c70d606/volumes" Jul 15 11:06:22.767229 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:22.767185 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:06:22.768984 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:22.768964 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:06:27.867554 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:27.867516 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78"] Jul 15 11:06:27.868348 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:27.867724 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" containerID="cri-o://a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843" gracePeriod=30 Jul 15 11:06:27.868764 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:27.868663 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:28.112151 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:28.112113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:28.218290 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:28.218210 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:28.316897 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:28.316852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:28.417388 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:28.417359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:28.509628 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:28.509518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:28.592545 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:28.592509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:30.154212 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:30.154168 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:35.153877 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:35.153829 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:40.154185 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:40.154139 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:40.154623 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:40.154249 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 11:06:45.153953 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:45.153910 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:50.153796 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:50.153755 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:55.154489 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:55.154448 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:06:58.035354 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.035323 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 11:06:58.038040 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.038014 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn"] Jul 15 11:06:58.038371 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.038358 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" Jul 15 11:06:58.038371 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.038372 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" Jul 15 11:06:58.038489 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.038387 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" Jul 15 11:06:58.038489 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.038394 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" Jul 15 11:06:58.038489 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.038454 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerName="sequence-graph-074d2" Jul 15 11:06:58.038489 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.038461 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3e32df-59f4-4a0e-86b6-4e707c70d606" containerName="switch-graph-ff4f8" Jul 15 11:06:58.041185 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.041165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.044474 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.044452 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cf436-serving-cert\"" Jul 15 11:06:58.044623 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.044602 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cf436-kube-rbac-proxy-sar-config\"" Jul 15 11:06:58.046852 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.046823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:58.051238 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.051212 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn"] Jul 15 11:06:58.158844 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.158805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bfd496a-bb37-4b4a-bad2-c567178570e3-openshift-service-ca-bundle\") pod \"3bfd496a-bb37-4b4a-bad2-c567178570e3\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " Jul 15 11:06:58.158844 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.158848 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bfd496a-bb37-4b4a-bad2-c567178570e3-proxy-tls\") pod \"3bfd496a-bb37-4b4a-bad2-c567178570e3\" (UID: \"3bfd496a-bb37-4b4a-bad2-c567178570e3\") " Jul 15 11:06:58.159060 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.158948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls\") pod \"ensemble-graph-cf436-7d4dbbfd97-vqcsn\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.159060 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.159016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-openshift-service-ca-bundle\") pod \"ensemble-graph-cf436-7d4dbbfd97-vqcsn\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.159286 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.159256 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfd496a-bb37-4b4a-bad2-c567178570e3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3bfd496a-bb37-4b4a-bad2-c567178570e3" (UID: "3bfd496a-bb37-4b4a-bad2-c567178570e3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:06:58.160998 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.160973 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bfd496a-bb37-4b4a-bad2-c567178570e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bfd496a-bb37-4b4a-bad2-c567178570e3" (UID: "3bfd496a-bb37-4b4a-bad2-c567178570e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:06:58.260322 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.260280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-openshift-service-ca-bundle\") pod \"ensemble-graph-cf436-7d4dbbfd97-vqcsn\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.260502 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.260345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls\") pod \"ensemble-graph-cf436-7d4dbbfd97-vqcsn\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.260502 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.260380 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bfd496a-bb37-4b4a-bad2-c567178570e3-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:06:58.260502 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.260390 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bfd496a-bb37-4b4a-bad2-c567178570e3-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:06:58.260502 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:06:58.260463 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-cf436-serving-cert: secret "ensemble-graph-cf436-serving-cert" not found Jul 15 11:06:58.260669 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:06:58.260530 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls podName:bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3 nodeName:}" failed. No retries permitted until 2026-07-15 11:06:58.760513135 +0000 UTC m=+1236.536964351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls") pod "ensemble-graph-cf436-7d4dbbfd97-vqcsn" (UID: "bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3") : secret "ensemble-graph-cf436-serving-cert" not found Jul 15 11:06:58.260974 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.260954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-openshift-service-ca-bundle\") pod \"ensemble-graph-cf436-7d4dbbfd97-vqcsn\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.523014 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.522965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:06:58.547107 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.547069 2580 generic.go:356] "Generic (PLEG): container finished" podID="3bfd496a-bb37-4b4a-bad2-c567178570e3" containerID="a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843" exitCode=137 Jul 15 11:06:58.547107 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.547107 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" Jul 15 11:06:58.547340 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.547135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" event={"ID":"3bfd496a-bb37-4b4a-bad2-c567178570e3","Type":"ContainerDied","Data":"a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843"} Jul 15 11:06:58.547340 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.547185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78" event={"ID":"3bfd496a-bb37-4b4a-bad2-c567178570e3","Type":"ContainerDied","Data":"a8376c46b49112c72f2fb0a62eb3b979e2c7a89163d3e5029277631afaa1143a"} Jul 15 11:06:58.547340 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.547210 2580 scope.go:117] "RemoveContainer" containerID="a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843" Jul 15 11:06:58.556130 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.556112 2580 scope.go:117] "RemoveContainer" containerID="a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843" Jul 15 11:06:58.556402 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:06:58.556383 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843\": container with ID starting with a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843 not found: ID does not exist" containerID="a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843" Jul 15 11:06:58.556458 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.556416 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843"} err="failed to get container status \"a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843\": rpc error: code = NotFound desc = could not find container \"a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843\": container with ID starting with a1117c368e548ec627480aa6df6b8e9bdbb4848ddfdbc036bdaa06d095734843 not found: ID does not exist" Jul 15 11:06:58.569120 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.569063 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78"] Jul 15 11:06:58.571317 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.571294 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78"] Jul 15 11:06:58.764174 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.764141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls\") pod \"ensemble-graph-cf436-7d4dbbfd97-vqcsn\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.766702 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.766678 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls\") pod \"ensemble-graph-cf436-7d4dbbfd97-vqcsn\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:58.825312 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.825227 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bfd496a-bb37-4b4a-bad2-c567178570e3" path="/var/lib/kubelet/pods/3bfd496a-bb37-4b4a-bad2-c567178570e3/volumes" Jul 15 11:06:58.953322 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:58.953283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:59.096192 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:59.096156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn"] Jul 15 11:06:59.099995 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:06:59.099962 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4f73ea_b479_4bfa_a4b3_5d4dc3a89ad3.slice/crio-5270e25954e98ca2701b0f54288c79e6eba6ad773c0b26e74bd1a1db96abbb3f WatchSource:0}: Error finding container 5270e25954e98ca2701b0f54288c79e6eba6ad773c0b26e74bd1a1db96abbb3f: Status 404 returned error can't find the container with id 5270e25954e98ca2701b0f54288c79e6eba6ad773c0b26e74bd1a1db96abbb3f Jul 15 11:06:59.553530 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:59.553433 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" event={"ID":"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3","Type":"ContainerStarted","Data":"656212d4a21702914cb08e6fde57e441ae928ffc1b89eb9f8fbdfe3f7f7102d3"} Jul 15 11:06:59.553530 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:59.553494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" event={"ID":"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3","Type":"ContainerStarted","Data":"5270e25954e98ca2701b0f54288c79e6eba6ad773c0b26e74bd1a1db96abbb3f"} Jul 15 11:06:59.553779 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:59.553547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:06:59.574434 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:06:59.574377 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podStartSLOduration=1.574360515 podStartE2EDuration="1.574360515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:06:59.572490754 +0000 UTC m=+1237.348942032" watchObservedRunningTime="2026-07-15 11:06:59.574360515 +0000 UTC m=+1237.350811751" Jul 15 11:07:05.565250 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:05.565220 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:07:08.101214 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:08.101176 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn"] Jul 15 11:07:08.101720 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:08.101410 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" containerID="cri-o://656212d4a21702914cb08e6fde57e441ae928ffc1b89eb9f8fbdfe3f7f7102d3" gracePeriod=30 Jul 15 11:07:08.127177 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:08.127139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:08.236218 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:08.236187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:08.331927 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:08.331885 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:08.409232 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:08.409105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:08.481863 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:08.481827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:09.131243 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:09.131211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:10.564070 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:10.564029 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:07:15.563240 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:15.563142 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:07:20.563569 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:20.563521 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:07:20.564004 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:20.563678 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:07:25.563707 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:25.563668 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:07:30.563613 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:30.563557 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:07:35.563194 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:35.563151 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:07:38.101978 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.101942 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc"] Jul 15 11:07:38.106855 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.106832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.109442 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.109414 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-30120-kube-rbac-proxy-sar-config\"" Jul 15 11:07:38.109645 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.109624 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-30120-serving-cert\"" Jul 15 11:07:38.117866 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.117833 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc"] Jul 15 11:07:38.137136 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.137100 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:38.194656 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.194617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls\") pod \"sequence-graph-30120-7b65c6b7f-q2rbc\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.194846 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.194677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c17d742e-9485-43cf-bf97-048f41f53351-openshift-service-ca-bundle\") pod \"sequence-graph-30120-7b65c6b7f-q2rbc\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.295953 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.295915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls\") pod \"sequence-graph-30120-7b65c6b7f-q2rbc\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.296146 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.295972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c17d742e-9485-43cf-bf97-048f41f53351-openshift-service-ca-bundle\") pod \"sequence-graph-30120-7b65c6b7f-q2rbc\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.296146 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:07:38.296097 2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-30120-serving-cert: secret "sequence-graph-30120-serving-cert" not found Jul 15 11:07:38.296261 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:07:38.296173 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls podName:c17d742e-9485-43cf-bf97-048f41f53351 nodeName:}" failed. No retries permitted until 2026-07-15 11:07:38.79615234 +0000 UTC m=+1276.572603555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls") pod "sequence-graph-30120-7b65c6b7f-q2rbc" (UID: "c17d742e-9485-43cf-bf97-048f41f53351") : secret "sequence-graph-30120-serving-cert" not found Jul 15 11:07:38.296610 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.296593 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c17d742e-9485-43cf-bf97-048f41f53351-openshift-service-ca-bundle\") pod \"sequence-graph-30120-7b65c6b7f-q2rbc\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.550039 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.549931 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:38.749487 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.749452 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerID="656212d4a21702914cb08e6fde57e441ae928ffc1b89eb9f8fbdfe3f7f7102d3" exitCode=0 Jul 15 11:07:38.749661 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.749521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" event={"ID":"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3","Type":"ContainerDied","Data":"656212d4a21702914cb08e6fde57e441ae928ffc1b89eb9f8fbdfe3f7f7102d3"} Jul 15 11:07:38.749661 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.749612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" event={"ID":"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3","Type":"ContainerDied","Data":"5270e25954e98ca2701b0f54288c79e6eba6ad773c0b26e74bd1a1db96abbb3f"} Jul 15 11:07:38.749661 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.749628 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5270e25954e98ca2701b0f54288c79e6eba6ad773c0b26e74bd1a1db96abbb3f" Jul 15 11:07:38.751469 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.751449 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:07:38.799834 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.799798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-openshift-service-ca-bundle\") pod \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " Jul 15 11:07:38.800036 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.799856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls\") pod \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\" (UID: \"bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3\") " Jul 15 11:07:38.800131 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.800082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls\") pod \"sequence-graph-30120-7b65c6b7f-q2rbc\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.800196 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.800141 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" (UID: "bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:07:38.802070 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.802040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" (UID: "bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:07:38.802357 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.802340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls\") pod \"sequence-graph-30120-7b65c6b7f-q2rbc\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:38.900949 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.900905 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:07:38.900949 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:38.900938 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:07:39.069127 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.069025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:39.205439 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.205415 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc"] Jul 15 11:07:39.207675 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:07:39.207646 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17d742e_9485_43cf_bf97_048f41f53351.slice/crio-33747ec6ce3a48c1adaad4be46d19643d6d93da7251ecbc3fa20e748aa685674 WatchSource:0}: Error finding container 33747ec6ce3a48c1adaad4be46d19643d6d93da7251ecbc3fa20e748aa685674: Status 404 returned error can't find the container with id 33747ec6ce3a48c1adaad4be46d19643d6d93da7251ecbc3fa20e748aa685674 Jul 15 11:07:39.755519 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.755478 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" event={"ID":"c17d742e-9485-43cf-bf97-048f41f53351","Type":"ContainerStarted","Data":"d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733"} Jul 15 11:07:39.755736 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.755539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" event={"ID":"c17d742e-9485-43cf-bf97-048f41f53351","Type":"ContainerStarted","Data":"33747ec6ce3a48c1adaad4be46d19643d6d93da7251ecbc3fa20e748aa685674"} Jul 15 11:07:39.755736 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.755539 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn" Jul 15 11:07:39.755736 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.755583 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:39.775718 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.775664 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podStartSLOduration=1.7756475520000001 podStartE2EDuration="1.775647552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:39.775144705 +0000 UTC m=+1277.551595960" watchObservedRunningTime="2026-07-15 11:07:39.775647552 +0000 UTC m=+1277.552098789" Jul 15 11:07:39.790332 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.790296 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn"] Jul 15 11:07:39.793525 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:39.793493 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn"] Jul 15 11:07:40.824964 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:40.824929 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" path="/var/lib/kubelet/pods/bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3/volumes" Jul 15 11:07:45.766874 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:45.766847 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:07:48.158173 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:48.158137 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc"] Jul 15 11:07:48.158563 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:48.158361 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" containerID="cri-o://d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733" gracePeriod=30 Jul 15 11:07:48.493065 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:48.492951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:48.586224 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:48.586154 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:48.723409 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:48.723372 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:49.121437 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:49.121382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:07:50.764694 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:50.764651 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:07:55.764789 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:07:55.764750 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:08:00.764446 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:00.764396 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:08:00.764942 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:00.764544 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:08:05.764435 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:05.764394 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:08:08.405059 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.405015 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7"] Jul 15 11:08:08.405450 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.405383 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" Jul 15 11:08:08.405450 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.405394 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" Jul 15 11:08:08.405450 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.405448 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4f73ea-b479-4bfa-a4b3-5d4dc3a89ad3" containerName="ensemble-graph-cf436" Jul 15 11:08:08.408428 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.408407 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.411091 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.411064 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d6666-kube-rbac-proxy-sar-config\"" Jul 15 11:08:08.411091 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.411071 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d6666-serving-cert\"" Jul 15 11:08:08.413444 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.413399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:08:08.418149 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.418122 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7"] Jul 15 11:08:08.557925 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.557889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-openshift-service-ca-bundle\") pod \"ensemble-graph-d6666-6bd466db88-jkkg7\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.558106 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.557946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-proxy-tls\") pod \"ensemble-graph-d6666-6bd466db88-jkkg7\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.588807 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.588769 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:08:08.659356 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.659282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-openshift-service-ca-bundle\") pod \"ensemble-graph-d6666-6bd466db88-jkkg7\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.659356 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.659348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-proxy-tls\") pod \"ensemble-graph-d6666-6bd466db88-jkkg7\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.660120 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.660091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-openshift-service-ca-bundle\") pod \"ensemble-graph-d6666-6bd466db88-jkkg7\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.661733 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.661715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-proxy-tls\") pod \"ensemble-graph-d6666-6bd466db88-jkkg7\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.719783 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.719749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:08.860993 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.860968 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7"] Jul 15 11:08:08.863747 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:08:08.863711 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f2ac51_b8f5_49f8_8e53_18ac6f78f7d5.slice/crio-764b8c33baa037a0895e89b4b694473b2acdb939af3e3cc29700f599a55f7411 WatchSource:0}: Error finding container 764b8c33baa037a0895e89b4b694473b2acdb939af3e3cc29700f599a55f7411: Status 404 returned error can't find the container with id 764b8c33baa037a0895e89b4b694473b2acdb939af3e3cc29700f599a55f7411 Jul 15 11:08:08.889181 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:08.889150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" event={"ID":"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5","Type":"ContainerStarted","Data":"764b8c33baa037a0895e89b4b694473b2acdb939af3e3cc29700f599a55f7411"} Jul 15 11:08:09.895027 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:09.894992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" event={"ID":"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5","Type":"ContainerStarted","Data":"cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd"} Jul 15 11:08:09.895466 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:09.895057 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:09.915119 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:09.915070 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podStartSLOduration=1.91505426 podStartE2EDuration="1.91505426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:08:09.914455002 +0000 UTC m=+1307.690906265" watchObservedRunningTime="2026-07-15 11:08:09.91505426 +0000 UTC m=+1307.691505519" Jul 15 11:08:10.764490 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:10.764454 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:08:15.765140 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:15.765096 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:08:15.904774 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:15.904738 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:08:18.337055 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.337027 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:08:18.437394 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.437293 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls\") pod \"c17d742e-9485-43cf-bf97-048f41f53351\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " Jul 15 11:08:18.437394 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.437389 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c17d742e-9485-43cf-bf97-048f41f53351-openshift-service-ca-bundle\") pod \"c17d742e-9485-43cf-bf97-048f41f53351\" (UID: \"c17d742e-9485-43cf-bf97-048f41f53351\") " Jul 15 11:08:18.437795 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.437772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17d742e-9485-43cf-bf97-048f41f53351-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c17d742e-9485-43cf-bf97-048f41f53351" (UID: "c17d742e-9485-43cf-bf97-048f41f53351"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:08:18.439467 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.439448 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c17d742e-9485-43cf-bf97-048f41f53351" (UID: "c17d742e-9485-43cf-bf97-048f41f53351"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:08:18.538725 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.538676 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c17d742e-9485-43cf-bf97-048f41f53351-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:08:18.538725 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.538716 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c17d742e-9485-43cf-bf97-048f41f53351-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:08:18.936010 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.935976 2580 generic.go:356] "Generic (PLEG): container finished" podID="c17d742e-9485-43cf-bf97-048f41f53351" containerID="d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733" exitCode=0 Jul 15 11:08:18.936202 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.936017 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" Jul 15 11:08:18.936202 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.936018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" event={"ID":"c17d742e-9485-43cf-bf97-048f41f53351","Type":"ContainerDied","Data":"d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733"} Jul 15 11:08:18.936202 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.936131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc" event={"ID":"c17d742e-9485-43cf-bf97-048f41f53351","Type":"ContainerDied","Data":"33747ec6ce3a48c1adaad4be46d19643d6d93da7251ecbc3fa20e748aa685674"} Jul 15 11:08:18.936202 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.936161 2580 scope.go:117] "RemoveContainer" containerID="d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733" Jul 15 11:08:18.947663 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.947628 2580 scope.go:117] "RemoveContainer" containerID="d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733" Jul 15 11:08:18.948044 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:08:18.948017 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733\": container with ID starting with d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733 not found: ID does not exist" containerID="d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733" Jul 15 11:08:18.948114 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.948052 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733"} err="failed to get container status \"d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733\": rpc error: code = NotFound desc = could not find container \"d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733\": container with ID starting with d829594987652b21a56d79151e135bfa44f12fc06e9e503a4ac58a3242bb9733 not found: ID does not exist" Jul 15 11:08:18.963110 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.963071 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc"] Jul 15 11:08:18.969534 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:18.969505 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc"] Jul 15 11:08:20.825864 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:20.825826 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c17d742e-9485-43cf-bf97-048f41f53351" path="/var/lib/kubelet/pods/c17d742e-9485-43cf-bf97-048f41f53351/volumes" Jul 15 11:08:48.404828 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.404729 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:08:48.405872 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.405848 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh"] Jul 15 11:08:48.406243 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.406228 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" Jul 15 11:08:48.406243 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.406244 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" Jul 15 11:08:48.406329 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.406299 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17d742e-9485-43cf-bf97-048f41f53351" containerName="sequence-graph-30120" Jul 15 11:08:48.410729 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.410713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.413369 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.413348 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-70137-kube-rbac-proxy-sar-config\"" Jul 15 11:08:48.413484 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.413348 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-70137-serving-cert\"" Jul 15 11:08:48.418293 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.418027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh"] Jul 15 11:08:48.504935 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.504901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-openshift-service-ca-bundle\") pod \"sequence-graph-70137-5dcc4d4f8-8llvh\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.504935 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.504949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-proxy-tls\") pod \"sequence-graph-70137-5dcc4d4f8-8llvh\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.605819 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.605765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-openshift-service-ca-bundle\") pod \"sequence-graph-70137-5dcc4d4f8-8llvh\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.605819 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.605823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-proxy-tls\") pod \"sequence-graph-70137-5dcc4d4f8-8llvh\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.606430 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.606411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-openshift-service-ca-bundle\") pod \"sequence-graph-70137-5dcc4d4f8-8llvh\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.608288 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.608263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-proxy-tls\") pod \"sequence-graph-70137-5dcc4d4f8-8llvh\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.722287 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.722191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:48.861701 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:48.861668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh"] Jul 15 11:08:48.864730 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:08:48.864702 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d4b7a8_b3b6_47d7_afda_c02c919415c9.slice/crio-d32bb202b570ae20699a22ba89d506eecdc04457d6bf9028d5da88830961e7fb WatchSource:0}: Error finding container d32bb202b570ae20699a22ba89d506eecdc04457d6bf9028d5da88830961e7fb: Status 404 returned error can't find the container with id d32bb202b570ae20699a22ba89d506eecdc04457d6bf9028d5da88830961e7fb Jul 15 11:08:49.080526 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:49.080440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" event={"ID":"f8d4b7a8-b3b6-47d7-afda-c02c919415c9","Type":"ContainerStarted","Data":"29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd"} Jul 15 11:08:49.080526 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:49.080494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" event={"ID":"f8d4b7a8-b3b6-47d7-afda-c02c919415c9","Type":"ContainerStarted","Data":"d32bb202b570ae20699a22ba89d506eecdc04457d6bf9028d5da88830961e7fb"} Jul 15 11:08:49.080759 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:49.080556 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:08:49.099119 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:49.099067 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podStartSLOduration=1.099051935 podStartE2EDuration="1.099051935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:08:49.098475479 +0000 UTC m=+1346.874926765" watchObservedRunningTime="2026-07-15 11:08:49.099051935 +0000 UTC m=+1346.875503172" Jul 15 11:08:49.807815 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:49.807782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:08:55.091620 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:08:55.091565 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:09:36.205659 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:09:36.205611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:09:55.988160 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:09:55.988127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:11:22.802090 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:11:22.802061 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:11:22.802638 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:11:22.802330 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:11:36.231562 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:11:36.231525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:13:09.968696 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:13:09.968658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:13:22.858924 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:13:22.858845 2580 scope.go:117] "RemoveContainer" containerID="656212d4a21702914cb08e6fde57e441ae928ffc1b89eb9f8fbdfe3f7f7102d3" Jul 15 11:13:28.046778 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:13:28.046742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:16:22.835058 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:22.835015 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:16:22.835736 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:22.835719 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:16:22.919118 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:22.919082 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7"] Jul 15 11:16:22.919373 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:22.919325 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" containerID="cri-o://cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd" gracePeriod=30 Jul 15 11:16:22.965522 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:22.965477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:16:24.222789 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:24.222752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:16:25.903665 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:25.903621 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:16:30.903254 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:30.903214 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:16:35.903325 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:35.903284 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:16:35.903797 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:35.903429 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:16:40.903665 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:40.903623 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:16:45.903918 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:45.903872 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:16:50.903658 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:50.903619 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:16:53.078120 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.078081 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:16:53.236078 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.235976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-openshift-service-ca-bundle\") pod \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " Jul 15 11:16:53.236078 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.236075 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-proxy-tls\") pod \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\" (UID: \"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5\") " Jul 15 11:16:53.236402 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.236382 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" (UID: "61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:16:53.238228 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.238207 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" (UID: "61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:16:53.337546 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.337494 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:16:53.337546 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.337538 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:16:53.349618 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.349560 2580 generic.go:356] "Generic (PLEG): container finished" podID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerID="cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd" exitCode=0 Jul 15 11:16:53.349788 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.349617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" event={"ID":"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5","Type":"ContainerDied","Data":"cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd"} Jul 15 11:16:53.349788 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.349669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" event={"ID":"61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5","Type":"ContainerDied","Data":"764b8c33baa037a0895e89b4b694473b2acdb939af3e3cc29700f599a55f7411"} Jul 15 11:16:53.349788 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.349690 2580 scope.go:117] "RemoveContainer" containerID="cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd" Jul 15 11:16:53.349788 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.349623 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7" Jul 15 11:16:53.360610 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.360568 2580 scope.go:117] "RemoveContainer" containerID="cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd" Jul 15 11:16:53.361085 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:16:53.361057 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd\": container with ID starting with cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd not found: ID does not exist" containerID="cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd" Jul 15 11:16:53.361181 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.361098 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd"} err="failed to get container status \"cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd\": rpc error: code = NotFound desc = could not find container \"cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd\": container with ID starting with cb2016a176b8f126c0f41ea7edf1701fe10f1178ef2f83da8c4be334252bb7cd not found: ID does not exist" Jul 15 11:16:53.371683 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.371651 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7"] Jul 15 11:16:53.376126 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:53.376100 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7"] Jul 15 11:16:54.825975 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:16:54.825940 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" path="/var/lib/kubelet/pods/61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5/volumes" Jul 15 11:17:03.053452 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.053417 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh"] Jul 15 11:17:03.054014 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.053624 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" containerID="cri-o://29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd" gracePeriod=30 Jul 15 11:17:03.075001 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.074965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:03.162048 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.162008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:03.220374 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.220339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:03.283438 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.283397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:03.351274 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.351240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:03.444314 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.444272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:03.545384 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:03.545345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:05.089809 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:05.089765 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:10.090347 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:10.090300 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:15.089591 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:15.089537 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:15.090000 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:15.089668 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:17:16.256632 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:16.256566 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:20.089491 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:20.089451 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:23.194549 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.194460 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk"] Jul 15 11:17:23.194972 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.194838 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" Jul 15 11:17:23.194972 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.194850 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" Jul 15 11:17:23.194972 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.194922 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f2ac51-b8f5-49f8-8e53-18ac6f78f7d5" containerName="ensemble-graph-d6666" Jul 15 11:17:23.198110 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.198092 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.200784 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.200760 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-cdb7a-serving-cert\"" Jul 15 11:17:23.200919 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.200761 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-cdb7a-kube-rbac-proxy-sar-config\"" Jul 15 11:17:23.206467 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.206443 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk"] Jul 15 11:17:23.208547 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.208498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:23.301723 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.301681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cff83e-bc5d-458e-839a-a907efbf1670-proxy-tls\") pod \"splitter-graph-cdb7a-ffd5b4495-dstlk\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.301894 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.301811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8cff83e-bc5d-458e-839a-a907efbf1670-openshift-service-ca-bundle\") pod \"splitter-graph-cdb7a-ffd5b4495-dstlk\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.403248 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.403204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8cff83e-bc5d-458e-839a-a907efbf1670-openshift-service-ca-bundle\") pod \"splitter-graph-cdb7a-ffd5b4495-dstlk\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.403425 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.403258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cff83e-bc5d-458e-839a-a907efbf1670-proxy-tls\") pod \"splitter-graph-cdb7a-ffd5b4495-dstlk\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.403924 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.403895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8cff83e-bc5d-458e-839a-a907efbf1670-openshift-service-ca-bundle\") pod \"splitter-graph-cdb7a-ffd5b4495-dstlk\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.405692 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.405665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cff83e-bc5d-458e-839a-a907efbf1670-proxy-tls\") pod \"splitter-graph-cdb7a-ffd5b4495-dstlk\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.417078 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.417044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:23.509749 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.509660 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:23.646471 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:23.646443 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk"] Jul 15 11:17:23.648896 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:17:23.648856 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cff83e_bc5d_458e_839a_a907efbf1670.slice/crio-69f8e0f72235c42bcc22baffaacc42b3eaac4b936f595178cfe90abf8560f4bb WatchSource:0}: Error finding container 69f8e0f72235c42bcc22baffaacc42b3eaac4b936f595178cfe90abf8560f4bb: Status 404 returned error can't find the container with id 69f8e0f72235c42bcc22baffaacc42b3eaac4b936f595178cfe90abf8560f4bb Jul 15 11:17:24.487008 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:24.486963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" event={"ID":"a8cff83e-bc5d-458e-839a-a907efbf1670","Type":"ContainerStarted","Data":"34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3"} Jul 15 11:17:24.487473 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:24.487027 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:24.487473 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:24.487044 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" event={"ID":"a8cff83e-bc5d-458e-839a-a907efbf1670","Type":"ContainerStarted","Data":"69f8e0f72235c42bcc22baffaacc42b3eaac4b936f595178cfe90abf8560f4bb"} Jul 15 11:17:24.504941 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:24.504879 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podStartSLOduration=1.504858203 podStartE2EDuration="1.504858203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:17:24.504357491 +0000 UTC m=+1862.280808729" watchObservedRunningTime="2026-07-15 11:17:24.504858203 +0000 UTC m=+1862.281309440" Jul 15 11:17:25.089979 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:25.089940 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:30.090233 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:30.090190 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:30.498244 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:30.498163 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:33.210962 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.210931 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:17:33.274696 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.274653 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk"] Jul 15 11:17:33.274864 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.274847 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" containerID="cri-o://34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3" gracePeriod=30 Jul 15 11:17:33.282249 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.282201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:33.384461 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.384430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:33.394616 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.394566 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-openshift-service-ca-bundle\") pod \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " Jul 15 11:17:33.394787 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.394646 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-proxy-tls\") pod \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\" (UID: \"f8d4b7a8-b3b6-47d7-afda-c02c919415c9\") " Jul 15 11:17:33.395097 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.395070 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f8d4b7a8-b3b6-47d7-afda-c02c919415c9" (UID: "f8d4b7a8-b3b6-47d7-afda-c02c919415c9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:17:33.396929 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.396896 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8d4b7a8-b3b6-47d7-afda-c02c919415c9" (UID: "f8d4b7a8-b3b6-47d7-afda-c02c919415c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:17:33.450321 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.450224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:33.496090 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.496054 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:17:33.496090 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.496084 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8d4b7a8-b3b6-47d7-afda-c02c919415c9-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:17:33.529480 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.529449 2580 generic.go:356] "Generic (PLEG): container finished" podID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerID="29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd" exitCode=0 Jul 15 11:17:33.529670 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.529497 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" Jul 15 11:17:33.529670 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.529513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" event={"ID":"f8d4b7a8-b3b6-47d7-afda-c02c919415c9","Type":"ContainerDied","Data":"29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd"} Jul 15 11:17:33.529670 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.529556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh" event={"ID":"f8d4b7a8-b3b6-47d7-afda-c02c919415c9","Type":"ContainerDied","Data":"d32bb202b570ae20699a22ba89d506eecdc04457d6bf9028d5da88830961e7fb"} Jul 15 11:17:33.529670 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.529608 2580 scope.go:117] "RemoveContainer" containerID="29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd" Jul 15 11:17:33.539366 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.539346 2580 scope.go:117] "RemoveContainer" containerID="29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd" Jul 15 11:17:33.539727 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:17:33.539700 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd\": container with ID starting with 29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd not found: ID does not exist" containerID="29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd" Jul 15 11:17:33.539830 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.539740 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd"} err="failed to get container status \"29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd\": rpc error: code = NotFound desc = could not find container \"29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd\": container with ID starting with 29ccbb24f880b07fe1da22319a27d6f728c2e08a2233223aa834d40a548fc2cd not found: ID does not exist" Jul 15 11:17:33.565590 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.565546 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh"] Jul 15 11:17:33.572678 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.572639 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh"] Jul 15 11:17:33.583958 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.583925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:33.683046 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.683005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:33.983029 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:33.982993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:17:34.825447 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:34.825409 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" path="/var/lib/kubelet/pods/f8d4b7a8-b3b6-47d7-afda-c02c919415c9/volumes" Jul 15 11:17:35.496206 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:35.496161 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:40.496230 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:40.496190 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:45.495728 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:45.495631 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:45.496101 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:45.495732 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:17:50.495857 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:50.495816 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:17:55.495494 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:17:55.495443 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:18:00.495982 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:00.495940 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:18:03.314378 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.314340 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794"] Jul 15 11:18:03.314923 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.314733 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" Jul 15 11:18:03.314923 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.314746 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" Jul 15 11:18:03.314923 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.314820 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d4b7a8-b3b6-47d7-afda-c02c919415c9" containerName="sequence-graph-70137" Jul 15 11:18:03.317701 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.317672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.319196 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.319165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:18:03.319856 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.319836 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bc774-serving-cert\"" Jul 15 11:18:03.319955 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.319940 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bc774-kube-rbac-proxy-sar-config\"" Jul 15 11:18:03.325519 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.325494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794"] Jul 15 11:18:03.433711 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.433681 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:18:03.435327 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.435297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:18:03.461721 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.461690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-proxy-tls\") pod \"switch-graph-bc774-7778877fb9-g5794\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.461912 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.461823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-openshift-service-ca-bundle\") pod \"switch-graph-bc774-7778877fb9-g5794\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.563162 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.563060 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8cff83e-bc5d-458e-839a-a907efbf1670-openshift-service-ca-bundle\") pod \"a8cff83e-bc5d-458e-839a-a907efbf1670\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " Jul 15 11:18:03.563334 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.563167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cff83e-bc5d-458e-839a-a907efbf1670-proxy-tls\") pod \"a8cff83e-bc5d-458e-839a-a907efbf1670\" (UID: \"a8cff83e-bc5d-458e-839a-a907efbf1670\") " Jul 15 11:18:03.563334 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.563239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-proxy-tls\") pod \"switch-graph-bc774-7778877fb9-g5794\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.563334 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.563313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-openshift-service-ca-bundle\") pod \"switch-graph-bc774-7778877fb9-g5794\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.563604 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.563548 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cff83e-bc5d-458e-839a-a907efbf1670-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a8cff83e-bc5d-458e-839a-a907efbf1670" (UID: "a8cff83e-bc5d-458e-839a-a907efbf1670"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:18:03.563971 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.563947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-openshift-service-ca-bundle\") pod \"switch-graph-bc774-7778877fb9-g5794\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.565259 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.565234 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cff83e-bc5d-458e-839a-a907efbf1670-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8cff83e-bc5d-458e-839a-a907efbf1670" (UID: "a8cff83e-bc5d-458e-839a-a907efbf1670"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:18:03.565701 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.565683 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-proxy-tls\") pod \"switch-graph-bc774-7778877fb9-g5794\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.638133 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.638083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:03.664688 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.664652 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cff83e-bc5d-458e-839a-a907efbf1670-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:18:03.664688 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.664691 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8cff83e-bc5d-458e-839a-a907efbf1670-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:18:03.676036 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.675998 2580 generic.go:356] "Generic (PLEG): container finished" podID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerID="34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3" exitCode=0 Jul 15 11:18:03.676036 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.676038 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" Jul 15 11:18:03.676284 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.676077 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" event={"ID":"a8cff83e-bc5d-458e-839a-a907efbf1670","Type":"ContainerDied","Data":"34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3"} Jul 15 11:18:03.676284 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.676129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk" event={"ID":"a8cff83e-bc5d-458e-839a-a907efbf1670","Type":"ContainerDied","Data":"69f8e0f72235c42bcc22baffaacc42b3eaac4b936f595178cfe90abf8560f4bb"} Jul 15 11:18:03.676284 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.676151 2580 scope.go:117] "RemoveContainer" containerID="34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3" Jul 15 11:18:03.685949 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.685925 2580 scope.go:117] "RemoveContainer" containerID="34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3" Jul 15 11:18:03.686264 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:18:03.686246 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3\": container with ID starting with 34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3 not found: ID does not exist" containerID="34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3" Jul 15 11:18:03.686317 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.686273 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3"} err="failed to get container status \"34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3\": rpc error: code = NotFound desc = could not find container \"34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3\": container with ID starting with 34a6fce9878106ff31d8044bd3b3ceebe8489f7dbf0d130de182d946488891b3 not found: ID does not exist" Jul 15 11:18:03.699854 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.698635 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk"] Jul 15 11:18:03.701876 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.701836 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk"] Jul 15 11:18:03.778329 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:03.778212 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794"] Jul 15 11:18:03.781024 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:18:03.780999 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc0b3a5_fbf0_4cac_b4f8_be3a4d729942.slice/crio-824310e4b17bae9dfdb7e6fe708f025eaa535b5c51879d8adb4103925b4504a2 WatchSource:0}: Error finding container 824310e4b17bae9dfdb7e6fe708f025eaa535b5c51879d8adb4103925b4504a2: Status 404 returned error can't find the container with id 824310e4b17bae9dfdb7e6fe708f025eaa535b5c51879d8adb4103925b4504a2 Jul 15 11:18:04.681504 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:04.681466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" event={"ID":"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942","Type":"ContainerStarted","Data":"5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611"} Jul 15 11:18:04.681998 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:04.681562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" event={"ID":"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942","Type":"ContainerStarted","Data":"824310e4b17bae9dfdb7e6fe708f025eaa535b5c51879d8adb4103925b4504a2"} Jul 15 11:18:04.681998 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:04.681621 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:04.700341 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:04.700277 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podStartSLOduration=1.700262409 podStartE2EDuration="1.700262409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:18:04.698604137 +0000 UTC m=+1902.475055368" watchObservedRunningTime="2026-07-15 11:18:04.700262409 +0000 UTC m=+1902.476713645" Jul 15 11:18:04.825946 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:04.825906 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" path="/var/lib/kubelet/pods/a8cff83e-bc5d-458e-839a-a907efbf1670/volumes" Jul 15 11:18:10.693405 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:10.693371 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:18:33.524077 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.524042 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv"] Jul 15 11:18:33.524459 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.524440 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" Jul 15 11:18:33.524459 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.524456 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" Jul 15 11:18:33.524528 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.524518 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cff83e-bc5d-458e-839a-a907efbf1670" containerName="splitter-graph-cdb7a" Jul 15 11:18:33.527374 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.527343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:18:33.527518 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.527465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.529877 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.529854 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d1ffd-kube-rbac-proxy-sar-config\"" Jul 15 11:18:33.530033 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.529892 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-d1ffd-serving-cert\"" Jul 15 11:18:33.538025 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.537994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv"] Jul 15 11:18:33.619051 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.619013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-openshift-service-ca-bundle\") pod \"splitter-graph-d1ffd-7d9bf8b7bd-962wv\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.619051 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.619054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-proxy-tls\") pod \"splitter-graph-d1ffd-7d9bf8b7bd-962wv\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.719905 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.719869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-openshift-service-ca-bundle\") pod \"splitter-graph-d1ffd-7d9bf8b7bd-962wv\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.719905 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.719915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-proxy-tls\") pod \"splitter-graph-d1ffd-7d9bf8b7bd-962wv\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.720617 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.720593 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-openshift-service-ca-bundle\") pod \"splitter-graph-d1ffd-7d9bf8b7bd-962wv\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.722637 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.722601 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:18:33.722776 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.722758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-proxy-tls\") pod \"splitter-graph-d1ffd-7d9bf8b7bd-962wv\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.839589 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.839536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:33.984068 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:33.983984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv"] Jul 15 11:18:33.986557 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:18:33.986527 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc02e89b_ee61_4f1d_8c0c_8b69665e24ec.slice/crio-46e246f9337fe854fec85b7df9ac75f47c2a139d6bf2db0d603088fde6f2b463 WatchSource:0}: Error finding container 46e246f9337fe854fec85b7df9ac75f47c2a139d6bf2db0d603088fde6f2b463: Status 404 returned error can't find the container with id 46e246f9337fe854fec85b7df9ac75f47c2a139d6bf2db0d603088fde6f2b463 Jul 15 11:18:34.825318 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:34.825286 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:18:34.825712 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:34.825335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" event={"ID":"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec","Type":"ContainerStarted","Data":"76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae"} Jul 15 11:18:34.825712 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:34.825355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" event={"ID":"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec","Type":"ContainerStarted","Data":"46e246f9337fe854fec85b7df9ac75f47c2a139d6bf2db0d603088fde6f2b463"} Jul 15 11:18:34.844241 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:34.844185 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podStartSLOduration=1.844170372 podStartE2EDuration="1.844170372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:18:34.843285688 +0000 UTC m=+1932.619736926" watchObservedRunningTime="2026-07-15 11:18:34.844170372 +0000 UTC m=+1932.620621657" Jul 15 11:18:40.834372 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:18:40.834341 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:19:09.846742 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:19:09.846705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:20:10.259444 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:20:10.259412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:21:22.864419 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:21:22.864390 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:21:22.866325 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:21:22.866304 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:25:20.860914 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:25:20.860829 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:26:22.892060 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:22.892029 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:26:22.893912 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:22.893893 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:26:40.517724 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:40.517687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:26:48.209518 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:48.209437 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv"] Jul 15 11:26:48.209920 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:48.209669 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" containerID="cri-o://76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae" gracePeriod=30 Jul 15 11:26:48.254709 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:48.254672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:26:48.537386 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:48.537306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:26:48.661718 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:48.661686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:26:50.833086 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:50.833046 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:26:55.833052 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:26:55.833014 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:27:00.832402 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:00.832361 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:27:00.832825 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:00.832474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:27:05.833180 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:05.833144 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:27:10.832409 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:10.832372 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:27:15.832266 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:15.832229 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:27:18.365673 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:18.365646 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:27:18.510489 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:18.510397 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-openshift-service-ca-bundle\") pod \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " Jul 15 11:27:18.510687 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:18.510487 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-proxy-tls\") pod \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\" (UID: \"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec\") " Jul 15 11:27:18.510819 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:18.510793 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" (UID: "dc02e89b-ee61-4f1d-8c0c-8b69665e24ec"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:27:18.512558 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:18.512537 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" (UID: "dc02e89b-ee61-4f1d-8c0c-8b69665e24ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:27:18.611754 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:18.611719 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:27:18.611754 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:18.611748 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:27:19.226051 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.226014 2580 generic.go:356] "Generic (PLEG): container finished" podID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerID="76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae" exitCode=0 Jul 15 11:27:19.226251 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.226063 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" event={"ID":"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec","Type":"ContainerDied","Data":"76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae"} Jul 15 11:27:19.226251 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.226064 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" Jul 15 11:27:19.226251 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.226105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv" event={"ID":"dc02e89b-ee61-4f1d-8c0c-8b69665e24ec","Type":"ContainerDied","Data":"46e246f9337fe854fec85b7df9ac75f47c2a139d6bf2db0d603088fde6f2b463"} Jul 15 11:27:19.226251 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.226126 2580 scope.go:117] "RemoveContainer" containerID="76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae" Jul 15 11:27:19.234886 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.234868 2580 scope.go:117] "RemoveContainer" containerID="76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae" Jul 15 11:27:19.235144 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:27:19.235121 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae\": container with ID starting with 76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae not found: ID does not exist" containerID="76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae" Jul 15 11:27:19.235193 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.235157 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae"} err="failed to get container status \"76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae\": rpc error: code = NotFound desc = could not find container \"76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae\": container with ID starting with 76bf5c368eb29f89be4b39ff4696d135c0823ad327edecd7d3e6a0dcee6856ae not found: ID does not exist" Jul 15 11:27:19.255791 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.255759 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv"] Jul 15 11:27:19.270130 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:19.270100 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv"] Jul 15 11:27:20.825973 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:20.825938 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" path="/var/lib/kubelet/pods/dc02e89b-ee61-4f1d-8c0c-8b69665e24ec/volumes" Jul 15 11:27:39.767235 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:39.767198 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:27:50.329146 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:50.329106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:27:51.259220 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:27:51.259185 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:31:22.919691 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:31:22.919624 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:31:22.922056 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:31:22.921236 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:34:02.819222 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:02.819185 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:22.415457 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:22.415371 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794"] Jul 15 11:34:22.415891 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:22.415711 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" containerID="cri-o://5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611" gracePeriod=30 Jul 15 11:34:22.443132 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:22.443088 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:22.573749 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:22.573717 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:22.647895 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:22.647839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:24.180695 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:24.180656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:24.246115 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:24.246083 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:25.692079 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:25.692033 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:34:30.691134 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:30.691096 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:34:35.691419 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:35.691370 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:34:35.691946 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:35.691483 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:34:40.691718 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:40.691672 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:34:45.082420 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:45.082387 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:45.691374 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:45.691336 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:34:45.950625 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:45.950523 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:46.778368 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:46.778336 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:47.633096 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:47.633063 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:48.444704 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:48.444656 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:49.261221 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:49.261190 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:50.073256 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:50.073222 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:50.691655 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:50.691601 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 11:34:50.877721 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:50.877690 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:51.704054 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:51.704022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:52.560921 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.560895 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bc774-7778877fb9-g5794_8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/switch-graph-bc774/0.log" Jul 15 11:34:52.567964 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.567942 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:34:52.699340 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.699242 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-openshift-service-ca-bundle\") pod \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " Jul 15 11:34:52.699340 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.699320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-proxy-tls\") pod \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\" (UID: \"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942\") " Jul 15 11:34:52.699717 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.699694 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" (UID: "8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 11:34:52.701293 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.701277 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" (UID: "8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 11:34:52.800707 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.800654 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-openshift-service-ca-bundle\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:34:52.800707 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:52.800698 2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942-proxy-tls\") on node \"ip-10-0-130-140.ec2.internal\" DevicePath \"\"" Jul 15 11:34:53.236448 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.236415 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerID="5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611" exitCode=0 Jul 15 11:34:53.236643 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.236457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" event={"ID":"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942","Type":"ContainerDied","Data":"5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611"} Jul 15 11:34:53.236643 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.236511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" event={"ID":"8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942","Type":"ContainerDied","Data":"824310e4b17bae9dfdb7e6fe708f025eaa535b5c51879d8adb4103925b4504a2"} Jul 15 11:34:53.236643 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.236534 2580 scope.go:117] "RemoveContainer" containerID="5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611" Jul 15 11:34:53.236643 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.236461 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794" Jul 15 11:34:53.244842 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.244824 2580 scope.go:117] "RemoveContainer" containerID="5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611" Jul 15 11:34:53.245090 ip-10-0-130-140 kubenswrapper[2580]: E0715 11:34:53.245070 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611\": container with ID starting with 5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611 not found: ID does not exist" containerID="5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611" Jul 15 11:34:53.245156 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.245100 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611"} err="failed to get container status \"5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611\": rpc error: code = NotFound desc = could not find container \"5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611\": container with ID starting with 5e4869e15a68572e197d17fc3665bc9c1df56ba91d5c051dfba1d2abb06be611 not found: ID does not exist" Jul 15 11:34:53.252634 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.252612 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794"] Jul 15 11:34:53.257707 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:53.257687 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794"] Jul 15 11:34:54.825985 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:54.825953 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" path="/var/lib/kubelet/pods/8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942/volumes" Jul 15 11:34:58.171303 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:58.171267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:58.331286 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:58.331255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:34:58.475202 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:34:58.475111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:35:01.380186 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:01.380155 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d8q55_4f7beef7-c62f-468d-b02c-2a06d2322773/global-pull-secret-syncer/0.log" Jul 15 11:35:01.554103 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:01.554074 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" Jul 15 11:35:01.765528 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:01.765450 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-140.ec2.internal_04c4f391b82cdea4e8f49294bf51b41d/haproxy/0.log" Jul 15 11:35:03.910539 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:03.910505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:35:04.009244 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:04.009214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-s8dpc_ec45a64a-a53a-4025-93dd-938828aa0269/konnectivity-agent/0.log" limit=1024 Jul 15 11:35:05.731549 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:05.731517 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5608afc3-76eb-415c-87ec-c6ec07a25111/alertmanager/0.log" Jul 15 11:35:05.766357 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:05.766327 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5608afc3-76eb-415c-87ec-c6ec07a25111/config-reloader/0.log" Jul 15 11:35:05.797375 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:05.797348 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5608afc3-76eb-415c-87ec-c6ec07a25111/kube-rbac-proxy-web/0.log" Jul 15 11:35:05.826067 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:05.826045 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5608afc3-76eb-415c-87ec-c6ec07a25111/kube-rbac-proxy/0.log" Jul 15 11:35:05.863816 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:05.863789 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5608afc3-76eb-415c-87ec-c6ec07a25111/kube-rbac-proxy-metric/0.log" Jul 15 11:35:05.895809 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:05.895780 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5608afc3-76eb-415c-87ec-c6ec07a25111/prom-label-proxy/0.log" Jul 15 11:35:05.933079 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:05.933051 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5608afc3-76eb-415c-87ec-c6ec07a25111/init-config-reloader/0.log" Jul 15 11:35:06.164456 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:06.164426 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b6f74dfbc-lm4p4_a3956c90-8880-4430-a6af-d63809e5e7a5/metrics-server/0.log" Jul 15 11:35:06.425084 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:06.425010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnpf5_05b19dc3-60ee-4f6d-b758-8f2f51f6a548/node-exporter/0.log" Jul 15 11:35:06.465921 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:06.465899 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnpf5_05b19dc3-60ee-4f6d-b758-8f2f51f6a548/kube-rbac-proxy/0.log" Jul 15 11:35:06.502158 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:06.502130 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tnpf5_05b19dc3-60ee-4f6d-b758-8f2f51f6a548/init-textfile/0.log" Jul 15 11:35:06.535521 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:06.535483 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-mwl88_a69f4397-2b05-4e80-b5ab-56a8db3d3e2f/kube-rbac-proxy-main/0.log" Jul 15 11:35:06.566322 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:06.566298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-mwl88_a69f4397-2b05-4e80-b5ab-56a8db3d3e2f/kube-rbac-proxy-self/0.log" Jul 15 11:35:06.597551 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:06.597521 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-mwl88_a69f4397-2b05-4e80-b5ab-56a8db3d3e2f/openshift-state-metrics/0.log" Jul 15 11:35:07.077074 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.077049 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cb8458ff-6ws4t_70c6f363-2fd5-4643-a730-24519af9b8c0/telemeter-client/0.log" Jul 15 11:35:07.103224 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.103194 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cb8458ff-6ws4t_70c6f363-2fd5-4643-a730-24519af9b8c0/reload/0.log" Jul 15 11:35:07.128628 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.128601 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cb8458ff-6ws4t_70c6f363-2fd5-4643-a730-24519af9b8c0/kube-rbac-proxy/0.log" Jul 15 11:35:07.185533 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.185504 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578f4467db-f5ltg_614d32c4-4771-4134-9ef9-ce3233bbe6d5/thanos-query/0.log" Jul 15 11:35:07.231372 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.231344 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578f4467db-f5ltg_614d32c4-4771-4134-9ef9-ce3233bbe6d5/kube-rbac-proxy-web/0.log" Jul 15 11:35:07.276373 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.276345 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578f4467db-f5ltg_614d32c4-4771-4134-9ef9-ce3233bbe6d5/kube-rbac-proxy/0.log" Jul 15 11:35:07.314095 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.314068 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578f4467db-f5ltg_614d32c4-4771-4134-9ef9-ce3233bbe6d5/prom-label-proxy/0.log" Jul 15 11:35:07.374436 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.374402 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578f4467db-f5ltg_614d32c4-4771-4134-9ef9-ce3233bbe6d5/kube-rbac-proxy-rules/0.log" Jul 15 11:35:07.410392 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:07.410360 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578f4467db-f5ltg_614d32c4-4771-4134-9ef9-ce3233bbe6d5/kube-rbac-proxy-metrics/0.log" Jul 15 11:35:10.022324 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.022288 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj"] Jul 15 11:35:10.022714 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.022665 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" Jul 15 11:35:10.022714 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.022677 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" Jul 15 11:35:10.022714 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.022693 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" Jul 15 11:35:10.022714 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.022699 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" Jul 15 11:35:10.022848 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.022756 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc0b3a5-fbf0-4cac-b4f8-be3a4d729942" containerName="switch-graph-bc774" Jul 15 11:35:10.022848 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.022766 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc02e89b-ee61-4f1d-8c0c-8b69665e24ec" containerName="splitter-graph-d1ffd" Jul 15 11:35:10.025813 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.025789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.028240 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.028224 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f872n\"/\"default-dockercfg-tfb7w\"" Jul 15 11:35:10.028343 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.028245 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f872n\"/\"kube-root-ca.crt\"" Jul 15 11:35:10.029024 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.029005 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f872n\"/\"openshift-service-ca.crt\"" Jul 15 11:35:10.035774 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.035749 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj"] Jul 15 11:35:10.153347 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.153305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-lib-modules\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.153347 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.153349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-sys\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.153547 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.153366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244nd\" (UniqueName: \"kubernetes.io/projected/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-kube-api-access-244nd\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.153547 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.153444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-proc\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.153547 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.153476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-podres\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.254944 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.254910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-proc\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.254944 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.254951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-podres\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.255201 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.255039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-lib-modules\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.255201 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.255047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-proc\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.255201 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.255071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-sys\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.255201 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.255094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-244nd\" (UniqueName: \"kubernetes.io/projected/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-kube-api-access-244nd\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.255201 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.255151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-podres\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.255201 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.255162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-lib-modules\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.255201 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.255193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-sys\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.264387 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.264363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-244nd\" (UniqueName: \"kubernetes.io/projected/791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6-kube-api-access-244nd\") pod \"perf-node-gather-daemonset-8kxfj\" (UID: \"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6\") " pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.337068 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.337044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:10.469106 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:10.469056 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj"] Jul 15 11:35:10.471715 ip-10-0-130-140 kubenswrapper[2580]: W0715 11:35:10.471690 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod791924a8_e5f1_4b57_a85a_d7d5fbd0c0a6.slice/crio-649e7db4e76f837768d16c04e680155eb1b2f9a926636c453cd4c66f51086ad8 WatchSource:0}: Error finding container 649e7db4e76f837768d16c04e680155eb1b2f9a926636c453cd4c66f51086ad8: Status 404 returned error can't find the container with id 649e7db4e76f837768d16c04e680155eb1b2f9a926636c453cd4c66f51086ad8 Jul 15 11:35:11.137913 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.137872 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mz6kd_0668642b-02ed-4dbe-8081-abfc923974a5/dns/0.log" Jul 15 11:35:11.162188 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.162155 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mz6kd_0668642b-02ed-4dbe-8081-abfc923974a5/kube-rbac-proxy/0.log" Jul 15 11:35:11.289867 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.289838 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9npvb_a248225a-5a67-4ad2-9b38-fc3c04ee92ac/dns-node-resolver/0.log" Jul 15 11:35:11.318112 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.318079 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" event={"ID":"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6","Type":"ContainerStarted","Data":"1bd50b2bc636de840c1216869dca09c300914da306a05ce617b6c94ee542516a"} Jul 15 11:35:11.318287 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.318134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" event={"ID":"791924a8-e5f1-4b57-a85a-d7d5fbd0c0a6","Type":"ContainerStarted","Data":"649e7db4e76f837768d16c04e680155eb1b2f9a926636c453cd4c66f51086ad8"} Jul 15 11:35:11.318287 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.318158 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:11.334230 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.334184 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" podStartSLOduration=1.334169342 podStartE2EDuration="1.334169342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:35:11.332629717 +0000 UTC m=+2929.109080983" watchObservedRunningTime="2026-07-15 11:35:11.334169342 +0000 UTC m=+2929.110620579" Jul 15 11:35:11.806144 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.806116 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84c8f5d4dd-g7hxg_f48df869-59d0-472d-9b6a-a4df786659ae/registry/0.log" Jul 15 11:35:11.879116 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:11.879090 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j98bg_41649f74-e757-4ee4-9df0-a63755eb1b0c/node-ca/0.log" Jul 15 11:35:13.073458 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:13.073433 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tjz76_2d556996-8ed1-44c3-b41a-adf835e56a26/serve-healthcheck-canary/0.log" Jul 15 11:35:13.650314 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:13.650279 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ws2z8_ae568f44-9c3f-46fa-a508-22bea4aa54ae/kube-rbac-proxy/0.log" Jul 15 11:35:13.671727 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:13.671703 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ws2z8_ae568f44-9c3f-46fa-a508-22bea4aa54ae/exporter/0.log" Jul 15 11:35:13.694637 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:13.694592 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ws2z8_ae568f44-9c3f-46fa-a508-22bea4aa54ae/extractor/0.log" Jul 15 11:35:15.817923 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:15.817892 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-749567458d-kgznf_4df683d0-3a9c-431a-9a9e-cbe1d94f0690/manager/0.log" Jul 15 11:35:15.869500 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:15.869474 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-rtgm8_4c570fbb-952e-4fab-8dfe-4bf5acf78744/manager/0.log" Jul 15 11:35:15.920671 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:15.920647 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-55kc6_a8788539-8dca-4714-a3c1-48d9858ad9a0/server/0.log" Jul 15 11:35:16.027017 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:16.026987 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-xppm4_8e1162d4-4b2c-4fb6-a43f-658e2a9baa5d/manager/0.log" Jul 15 11:35:16.123951 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:16.123922 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-5wbmv_b49ebeb4-9383-48f3-8bc3-2882a1e30e02/seaweedfs/0.log" Jul 15 11:35:17.332363 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:17.332334 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f872n/perf-node-gather-daemonset-8kxfj" Jul 15 11:35:22.046186 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.046157 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwd87_40d807c3-969a-4305-a000-5eb13577cd18/kube-multus-additional-cni-plugins/0.log" Jul 15 11:35:22.067996 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.067968 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwd87_40d807c3-969a-4305-a000-5eb13577cd18/egress-router-binary-copy/0.log" Jul 15 11:35:22.088488 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.088455 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwd87_40d807c3-969a-4305-a000-5eb13577cd18/cni-plugins/0.log" Jul 15 11:35:22.109910 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.109885 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwd87_40d807c3-969a-4305-a000-5eb13577cd18/bond-cni-plugin/0.log" Jul 15 11:35:22.131087 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.131052 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwd87_40d807c3-969a-4305-a000-5eb13577cd18/routeoverride-cni/0.log" Jul 15 11:35:22.151268 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.151245 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwd87_40d807c3-969a-4305-a000-5eb13577cd18/whereabouts-cni-bincopy/0.log" Jul 15 11:35:22.170813 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.170776 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwd87_40d807c3-969a-4305-a000-5eb13577cd18/whereabouts-cni/0.log" Jul 15 11:35:22.396383 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.396343 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-slrhd_f26139f2-0422-42d7-b310-9cc1693f69ad/kube-multus/0.log" Jul 15 11:35:22.534441 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.534412 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mczbg_0119351c-bc27-4885-b458-ba76a4390657/network-metrics-daemon/0.log" Jul 15 11:35:22.552671 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:22.552643 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mczbg_0119351c-bc27-4885-b458-ba76a4390657/kube-rbac-proxy/0.log" Jul 15 11:35:23.272714 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.272680 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-controller/0.log" Jul 15 11:35:23.289453 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.289427 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/0.log" Jul 15 11:35:23.289453 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.289442 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovn-acl-logging/1.log" Jul 15 11:35:23.305943 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.305860 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/kube-rbac-proxy-node/0.log" Jul 15 11:35:23.331158 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.331127 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/kube-rbac-proxy-ovn-metrics/0.log" Jul 15 11:35:23.349959 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.349933 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/northd/0.log" Jul 15 11:35:23.376270 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.376214 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/nbdb/0.log" Jul 15 11:35:23.392089 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.392058 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/sbdb/0.log" Jul 15 11:35:23.434128 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:23.434096 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22ll8_f8874c1d-1f5f-4529-ac5b-bcec15627398/ovnkube-controller/0.log" Jul 15 11:35:24.863072 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:24.863041 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2fw2b_3439f5a7-5f74-422e-8089-c97186c19f74/network-check-target-container/0.log" Jul 15 11:35:25.848267 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:25.848239 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mqts8_8c5d1637-944f-41b0-9af0-d93f9b31c9a1/iptables-alerter/0.log" Jul 15 11:35:26.606558 ip-10-0-130-140 kubenswrapper[2580]: I0715 11:35:26.606511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-4tvsj_a6e6667c-5712-4706-943a-572aa76de3d3/tuned/0.log"