Jul 16 17:31:50.563658 ip-10-0-135-249 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 17:31:50.563670 ip-10-0-135-249 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 17:31:50.563676 ip-10-0-135-249 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 17:31:50.563882 ip-10-0-135-249 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 17:32:00.796749 ip-10-0-135-249 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 17:32:00.796765 ip-10-0-135-249 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b10e946cfd34489f8c4a535da1a1460d -- Jul 16 17:33:41.777086 ip-10-0-135-249 systemd[1]: Starting Kubernetes Kubelet... Jul 16 17:33:42.210657 ip-10-0-135-249 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:33:42.210657 ip-10-0-135-249 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 17:33:42.210657 ip-10-0-135-249 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:33:42.210657 ip-10-0-135-249 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 17:33:42.210657 ip-10-0-135-249 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:33:42.211963 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.211885 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 17:33:42.215818 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215803 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:33:42.215818 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215818 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215821 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215837 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215840 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215843 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215847 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215855 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215858 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215861 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215863 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215866 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215869 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215871 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215874 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215876 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215879 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215881 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215884 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215887 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215890 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:33:42.215887 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215892 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215895 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215898 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215901 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215903 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215906 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215908 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215911 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215913 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215916 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215918 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215920 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215923 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215926 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215928 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215931 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215933 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215936 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215939 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215943 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:33:42.216358 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215946 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215949 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215952 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215955 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215957 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215960 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215963 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215992 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215995 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.215997 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216000 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216003 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216006 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216008 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216013 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216017 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216020 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216023 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216025 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216028 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:33:42.216853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216031 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216035 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216039 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216042 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216044 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216047 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216049 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216052 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216054 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216058 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216061 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216064 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216066 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216069 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216071 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216074 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216076 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216079 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216082 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216084 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:33:42.217334 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216088 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216091 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216093 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216095 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216098 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216101 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216104 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216107 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216109 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216112 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216114 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216117 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216121 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216124 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216127 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216130 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216133 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216136 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216811 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:33:42.217800 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216816 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216819 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216822 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216846 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216850 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216852 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216856 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216858 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216861 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216864 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216866 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216869 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216872 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216875 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216877 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216880 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216882 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216885 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216887 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:33:42.218265 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216891 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216893 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216896 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216898 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216901 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216904 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216906 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216908 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216911 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216913 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216916 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216918 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216921 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216923 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216926 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216929 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216931 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216933 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216936 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216939 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:33:42.218722 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216942 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216944 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216947 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216950 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216952 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216955 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216958 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216960 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216963 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216965 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216967 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216971 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216973 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216978 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216981 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216984 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216987 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216990 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216993 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:33:42.219212 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216995 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.216999 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217002 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217005 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217008 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217010 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217013 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217016 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217019 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217021 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217024 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217026 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217029 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217032 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217035 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217038 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217041 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217043 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217046 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217049 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:33:42.219673 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217051 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217054 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217057 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217060 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217063 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217065 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217068 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217071 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217074 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217077 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217079 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217082 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217084 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217088 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217091 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217094 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217097 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217101 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217104 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:33:42.220174 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217107 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217460 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217464 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217468 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217471 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217474 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217476 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217479 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217482 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217485 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217488 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217490 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217493 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217496 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217498 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217501 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217503 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217506 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217509 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217511 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:33:42.220649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217514 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217516 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217519 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217522 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217524 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217526 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217529 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217532 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217534 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217537 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217539 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217541 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217544 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217546 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217549 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217551 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217554 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217557 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217561 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:33:42.221144 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217564 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217568 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217571 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217574 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217576 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217579 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217581 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217586 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217590 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217592 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217596 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217599 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217602 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217605 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217607 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217610 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217612 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217615 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217617 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:33:42.221599 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217620 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217622 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217625 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217627 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217630 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217632 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217635 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217638 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217640 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217643 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217645 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217647 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217650 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217653 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217656 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217658 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217661 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217663 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217666 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217668 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:33:42.222103 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217671 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217673 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217676 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217678 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217681 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217683 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217686 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217688 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217691 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217693 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217696 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217698 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217700 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217703 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217706 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217709 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217713 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217715 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217719 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:33:42.222578 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217722 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217725 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.217728 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217816 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217836 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217842 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217848 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217853 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217856 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217860 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217864 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217868 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217871 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217874 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217878 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217881 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217886 2564 flags.go:64] FLAG: --cgroup-root="" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217889 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217892 2564 flags.go:64] FLAG: --client-ca-file="" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217895 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217898 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217902 2564 flags.go:64] FLAG: --cluster-domain="" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217905 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217908 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 17:33:42.223080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217911 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217914 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217918 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217921 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217924 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217927 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217930 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217933 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217936 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217939 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217942 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217946 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217949 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217952 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217957 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217960 2564 flags.go:64] FLAG: --enable-server="true" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217963 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217967 2564 flags.go:64] FLAG: --event-burst="100" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217970 2564 flags.go:64] FLAG: --event-qps="50" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217973 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217976 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217979 2564 flags.go:64] FLAG: --eviction-hard="" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217982 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217985 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217990 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 17:33:42.223643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217993 2564 flags.go:64] FLAG: --eviction-soft="" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217996 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.217999 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218002 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218005 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218008 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218011 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218013 2564 flags.go:64] FLAG: --feature-gates="" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218017 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218020 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218023 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218027 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218030 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218032 2564 flags.go:64] FLAG: --help="false" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218035 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218038 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218041 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218044 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218048 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218051 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218054 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218058 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218061 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 17:33:42.224262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218064 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218067 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218071 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218074 2564 flags.go:64] FLAG: --kube-reserved="" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218077 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218079 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218082 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218085 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218090 2564 flags.go:64] FLAG: --lock-file="" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218093 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218096 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218099 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218104 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218107 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218110 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218113 2564 flags.go:64] FLAG: --logging-format="text" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218115 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218119 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218121 2564 flags.go:64] FLAG: --manifest-url="" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218124 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218128 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218131 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218135 2564 flags.go:64] FLAG: --max-pods="110" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218138 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218141 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 17:33:42.224855 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218144 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218146 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218150 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218152 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218155 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218164 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218167 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218170 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218173 2564 flags.go:64] FLAG: --pod-cidr="" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218176 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218181 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218184 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218187 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218190 2564 flags.go:64] FLAG: --port="10250" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218193 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218196 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c65cc579fb90445f" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218201 2564 flags.go:64] FLAG: --qos-reserved="" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218204 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218207 2564 flags.go:64] FLAG: --register-node="true" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218210 2564 flags.go:64] FLAG: --register-with-taints="" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218213 2564 flags.go:64] FLAG: --registry-burst="10" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218216 2564 flags.go:64] FLAG: --registry-qps="5" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218219 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218221 2564 flags.go:64] FLAG: --reserved-memory="" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218225 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 17:33:42.225459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218228 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218231 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218234 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218236 2564 flags.go:64] FLAG: --runonce="false" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218239 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218242 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218245 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218248 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218250 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218253 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218257 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218260 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218264 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218266 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218269 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218272 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218276 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218279 2564 flags.go:64] FLAG: --system-cgroups="" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218282 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218287 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218290 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218293 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218297 2564 flags.go:64] FLAG: --tls-min-version="" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218301 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218304 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 17:33:42.226072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218307 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218310 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218313 2564 flags.go:64] FLAG: --v="2" Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218316 2564 flags.go:64] FLAG: --version="false" Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218320 2564 flags.go:64] FLAG: --vmodule="" Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218324 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.218327 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218454 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218459 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218462 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218464 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218468 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218470 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218473 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218475 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218478 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218481 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218483 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218486 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218489 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218492 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218494 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:33:42.226696 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218497 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218499 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218502 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218505 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218507 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218510 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218512 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218515 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218519 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218521 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218524 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218526 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218529 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218531 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218534 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218537 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218539 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218542 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218544 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:33:42.227249 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218547 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218549 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218552 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218555 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218558 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218560 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218563 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218566 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218568 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218571 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218573 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218576 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218579 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218581 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218584 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218586 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218590 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218595 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218598 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:33:42.227720 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218600 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218604 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218608 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218611 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218614 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218617 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218619 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218622 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218625 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218628 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218631 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218633 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218636 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218639 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218641 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218644 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218647 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218650 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218652 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218655 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:33:42.228255 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218658 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218660 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218663 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218666 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218670 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218673 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218676 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218679 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218683 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218685 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218688 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218691 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218693 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218696 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218701 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218704 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218706 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218710 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218713 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:33:42.228823 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218717 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218720 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218722 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218725 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218728 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218731 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.218733 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.219438 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.225980 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.225994 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.226012 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.226018 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226072 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226077 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:33:42.229410 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226083 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226088 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226092 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226095 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226099 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226102 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226105 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226108 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226110 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226113 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226116 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226119 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226122 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226124 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226127 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226129 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226132 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226134 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226137 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:33:42.229758 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226139 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226142 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226145 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226147 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226150 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226152 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226155 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226158 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226161 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226164 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226167 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226169 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226172 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226175 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226177 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226180 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226182 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226185 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226188 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:33:42.230289 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226191 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226195 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226197 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226200 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226203 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226206 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226209 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226211 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226214 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226217 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226219 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226222 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226224 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226227 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226229 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226232 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226234 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226237 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226239 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226242 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:33:42.231071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226245 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226248 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226250 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226253 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226255 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226258 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226260 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226263 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226266 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226268 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226270 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226273 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226275 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226278 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226281 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226283 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226285 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226288 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226291 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226294 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:33:42.231552 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226296 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226299 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226301 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226304 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226306 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226308 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226311 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226313 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226316 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226318 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226321 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226324 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226326 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226329 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226332 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226334 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226337 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226339 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:33:42.232074 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226342 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.226347 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226449 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226453 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226457 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226460 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226462 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226465 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226469 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226472 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226475 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226478 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226481 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226484 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226486 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:33:42.232640 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226489 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226491 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226494 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226496 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226499 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226501 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226504 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226506 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226509 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226511 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226514 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226517 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226519 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226522 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226524 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226527 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226529 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226532 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226534 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:33:42.233071 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226537 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226539 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226542 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226544 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226546 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226549 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226551 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226554 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226556 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226559 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226561 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226564 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226568 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226570 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226572 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226575 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226577 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226579 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226582 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226584 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:33:42.233594 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226587 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226589 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226592 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226595 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226597 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226600 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226602 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226604 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226607 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226610 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226612 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226614 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226617 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226619 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226622 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226626 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226630 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226634 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226637 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226640 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:33:42.234098 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226642 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226645 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226647 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226649 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226652 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226655 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226657 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226660 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226662 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226664 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226667 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226669 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226672 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226674 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226677 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226679 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226682 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226684 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226687 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226689 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:33:42.234649 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226692 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226694 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226697 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226700 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226702 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226705 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:42.226708 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.226713 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.226801 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.231615 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.232964 2564 server.go:1019] "Starting client certificate rotation" Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.233072 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:33:42.235143 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.233112 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:33:42.236176 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.236154 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:33:42.237293 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.237259 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:33:42.244668 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.244652 2564 log.go:25] "Validated CRI v1 runtime API" Jul 16 17:33:42.248540 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.248515 2564 log.go:25] "Validated CRI v1 image API" Jul 16 17:33:42.249278 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.249263 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 17:33:42.251806 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.251788 2564 fs.go:135] Filesystem UUIDs: map[156c2839-7ded-406a-af5b-26ce943c7f8d:/dev/nvme0n1p4 241fdf59-6270-4abd-bf6f-e1c1f3a4ab73:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 16 17:33:42.251867 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.251807 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 17:33:42.258439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.258333 2564 manager.go:216] Machine: {Timestamp:2026-07-16 17:33:42.257046825 +0000 UTC m=+0.360150656 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112500 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d54d0b1f451753b7c08349e31de3f SystemUUID:ec2d54d0-b1f4-5175-3b7c-08349e31de3f BootID:b10e946c-fd34-489f-8c4a-535da1a1460d Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:05:e3:1a:97:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:05:e3:1a:97:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:f1:f9:3d:e5:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 17:33:42.258439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.258434 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 17:33:42.258559 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.258551 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 17:33:42.261850 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.261807 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 17:33:42.261989 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.261852 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-249.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 17:33:42.262636 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.262621 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:33:42.262739 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.262724 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 17:33:42.262773 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.262744 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 17:33:42.262773 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.262752 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:33:42.263489 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.263479 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:33:42.264746 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.264736 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 17:33:42.268317 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268305 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:33:42.268444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268434 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 17:33:42.268480 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268466 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 16 17:33:42.268480 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268478 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 17:33:42.268567 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268503 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 17:33:42.268567 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268518 2564 kubelet.go:398] "Adding apiserver pod source" Jul 16 17:33:42.268567 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268530 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 17:33:42.268983 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268971 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:33:42.269024 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.268990 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:33:42.273915 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.273889 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 17:33:42.275765 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275745 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 17:33:42.275878 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275778 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 17:33:42.275964 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275953 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 17:33:42.276015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275968 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 17:33:42.276015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275974 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 17:33:42.276015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275980 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 17:33:42.276015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275988 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 17:33:42.276015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.275993 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 17:33:42.276556 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.276547 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 17:33:42.276556 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.276555 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 17:33:42.276617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.276562 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 17:33:42.276617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.276572 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 17:33:42.276617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.276580 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 17:33:42.276617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.276587 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 17:33:42.278288 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.278273 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 17:33:42.278324 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.278291 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 17:33:42.278794 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.278774 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 17:33:42.278794 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.278777 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 17:33:42.280269 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.280251 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-249.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 17:33:42.281575 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.281562 2564 server.go:1268] "Started kubelet" Jul 16 17:33:42.281693 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.281644 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 17:33:42.281836 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.281641 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 17:33:42.281992 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.281974 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 17:33:42.282545 ip-10-0-135-249 systemd[1]: Started Kubernetes Kubelet. Jul 16 17:33:42.283386 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.283372 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 17:33:42.284058 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.284045 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 16 17:33:42.286847 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.286818 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 17:33:42.288185 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.288169 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 17:33:42.288185 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.288175 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 17:33:42.288331 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.288228 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.288692 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.287448 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-249.ec2.internal.18c2d5914417f6e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-249.ec2.internal,UID:ip-10-0-135-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-249.ec2.internal,},FirstTimestamp:2026-07-16 17:33:42.281533157 +0000 UTC m=+0.384636987,LastTimestamp:2026-07-16 17:33:42.281533157 +0000 UTC m=+0.384636987,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-249.ec2.internal,}" Jul 16 17:33:42.289058 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289042 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 17:33:42.289132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289060 2564 factory.go:55] Registering systemd factory Jul 16 17:33:42.289132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289072 2564 factory.go:223] Registration of the systemd container factory successfully Jul 16 17:33:42.289132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289114 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 17:33:42.289132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289115 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 17:33:42.289338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289135 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 17:33:42.289338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289295 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 16 17:33:42.289338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289305 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 16 17:33:42.289445 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289383 2564 factory.go:153] Registering CRI-O factory Jul 16 17:33:42.289445 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289399 2564 factory.go:223] Registration of the crio container factory successfully Jul 16 17:33:42.289445 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289420 2564 factory.go:103] Registering Raw factory Jul 16 17:33:42.289445 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289430 2564 manager.go:1196] Started watching for new ooms in manager Jul 16 17:33:42.289750 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.289737 2564 manager.go:349] Starting recovery of all containers Jul 16 17:33:42.294946 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.294920 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 17:33:42.294946 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.294934 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 17:33:42.298133 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.298101 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7tjgf" Jul 16 17:33:42.300796 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.300772 2564 manager.go:354] Recovery completed Jul 16 17:33:42.305975 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.305958 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:33:42.306547 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.306531 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7tjgf" Jul 16 17:33:42.307766 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.307752 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:33:42.307819 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.307780 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:33:42.307819 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.307790 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:33:42.308355 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.308342 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 17:33:42.308412 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.308355 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 17:33:42.308412 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.308370 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:33:42.308478 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.308344 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:33:42.309074 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.309059 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:33:42.309155 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.309089 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:33:42.309155 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.309104 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:33:42.309978 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.309913 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-249.ec2.internal.18c2d59145a845ed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-249.ec2.internal,UID:ip-10-0-135-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-249.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-249.ec2.internal,},FirstTimestamp:2026-07-16 17:33:42.307767789 +0000 UTC m=+0.410871622,LastTimestamp:2026-07-16 17:33:42.307767789 +0000 UTC m=+0.410871622,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-249.ec2.internal,}" Jul 16 17:33:42.310515 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.310503 2564 policy_none.go:49] "None policy: Start" Jul 16 17:33:42.310554 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.310518 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 17:33:42.310554 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.310529 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 17:33:42.312336 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.312325 2564 policy_none.go:47] "Start" Jul 16 17:33:42.356401 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.356384 2564 manager.go:337] "Starting Device Plugin manager" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.356416 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.356426 2564 server.go:87] "Starting device plugin registration server" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.356662 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.356672 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.356784 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.356890 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.356903 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.357421 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 17:33:42.377362 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.357470 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.457442 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.457393 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:33:42.458439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.458418 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:33:42.458537 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.458453 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:33:42.458537 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.458463 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:33:42.458537 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.458488 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.464979 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.464935 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.464979 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.464960 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-249.ec2.internal\": node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.479227 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.479206 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.566914 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.566873 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 17:33:42.568156 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.568139 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 17:33:42.568214 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.568177 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 17:33:42.568214 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.568203 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 17:33:42.568274 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.568250 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 17:33:42.570546 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.570522 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:33:42.580310 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.580286 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.668748 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.668698 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal"] Jul 16 17:33:42.668930 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.668790 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:33:42.672415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.672393 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:33:42.672532 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.672423 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:33:42.672532 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.672435 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:33:42.674701 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.674687 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:33:42.674818 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.674804 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.674897 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.674868 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:33:42.675401 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.675384 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:33:42.675474 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.675410 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:33:42.675474 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.675419 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:33:42.675474 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.675384 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:33:42.675564 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.675485 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:33:42.675564 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.675499 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:33:42.678004 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.677991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.678053 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.678017 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:33:42.678705 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.678688 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:33:42.678760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.678721 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:33:42.678760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.678736 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:33:42.680815 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.680798 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.690679 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.690660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be2e4781f1bc5622abf5feee16446d86-config\") pod \"kube-apiserver-proxy-ip-10-0-135-249.ec2.internal\" (UID: \"be2e4781f1bc5622abf5feee16446d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.690776 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.690683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3af81e36783efc72a4cd6203a0dae2b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal\" (UID: \"3af81e36783efc72a4cd6203a0dae2b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.690776 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.690702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af81e36783efc72a4cd6203a0dae2b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal\" (UID: \"3af81e36783efc72a4cd6203a0dae2b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.692382 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.692363 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-249.ec2.internal\" not found" node="ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.696754 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.696739 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-249.ec2.internal\" not found" node="ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.781484 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.781388 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.791799 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.791769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3af81e36783efc72a4cd6203a0dae2b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal\" (UID: \"3af81e36783efc72a4cd6203a0dae2b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.791915 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.791806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af81e36783efc72a4cd6203a0dae2b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal\" (UID: \"3af81e36783efc72a4cd6203a0dae2b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.791915 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.791847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be2e4781f1bc5622abf5feee16446d86-config\") pod \"kube-apiserver-proxy-ip-10-0-135-249.ec2.internal\" (UID: \"be2e4781f1bc5622abf5feee16446d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.791915 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.791885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3af81e36783efc72a4cd6203a0dae2b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal\" (UID: \"3af81e36783efc72a4cd6203a0dae2b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.791915 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.791904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be2e4781f1bc5622abf5feee16446d86-config\") pod \"kube-apiserver-proxy-ip-10-0-135-249.ec2.internal\" (UID: \"be2e4781f1bc5622abf5feee16446d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.792062 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.791885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af81e36783efc72a4cd6203a0dae2b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal\" (UID: \"3af81e36783efc72a4cd6203a0dae2b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.881980 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.881940 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.982502 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:42.982473 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:42.993932 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.993914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:42.999849 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:42.999819 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" Jul 16 17:33:43.082925 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.082822 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.116563 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.116538 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:33:43.183386 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.183351 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.232905 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.232877 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 17:33:43.233316 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.233019 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 17:33:43.233316 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.233056 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 17:33:43.283966 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.283946 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.310203 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.310166 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:28:42 +0000 UTC" deadline="2028-02-02 14:33:30.33120539 +0000 UTC" Jul 16 17:33:43.310203 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.310198 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13580h59m47.021010741s" Jul 16 17:33:43.384550 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.384487 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.485008 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.484984 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.585646 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.585625 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.591489 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:43.591456 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2e4781f1bc5622abf5feee16446d86.slice/crio-22141d71ce7076bf8b1ff40c8e075e79614340c611c8831f96f0826746bc1be7 WatchSource:0}: Error finding container 22141d71ce7076bf8b1ff40c8e075e79614340c611c8831f96f0826746bc1be7: Status 404 returned error can't find the container with id 22141d71ce7076bf8b1ff40c8e075e79614340c611c8831f96f0826746bc1be7 Jul 16 17:33:43.596910 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.596891 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:33:43.651766 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.651691 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:33:43.686625 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.686596 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.724194 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:43.724166 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af81e36783efc72a4cd6203a0dae2b4.slice/crio-6b4f8ddba1fb9317a42da990c82996529733c938785864a4a30eb1e7b04f436a WatchSource:0}: Error finding container 6b4f8ddba1fb9317a42da990c82996529733c938785864a4a30eb1e7b04f436a: Status 404 returned error can't find the container with id 6b4f8ddba1fb9317a42da990c82996529733c938785864a4a30eb1e7b04f436a Jul 16 17:33:43.787743 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:43.787694 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-249.ec2.internal\" not found" Jul 16 17:33:43.872686 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.872663 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:33:43.888477 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.888453 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" Jul 16 17:33:43.900041 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.900016 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:33:43.901141 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.901127 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" Jul 16 17:33:43.907852 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:43.907799 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:33:44.270144 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.270060 2564 apiserver.go:52] "Watching apiserver" Jul 16 17:33:44.276545 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.276518 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 17:33:44.278046 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.278023 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kq8gb","kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln","openshift-dns/node-resolver-wkxtr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal","openshift-multus/network-metrics-daemon-87b84","openshift-network-operator/iptables-alerter-glk67","kube-system/konnectivity-agent-rnkws","openshift-cluster-node-tuning-operator/tuned-tb7tc","openshift-image-registry/node-ca-lfccv","openshift-multus/multus-additional-cni-plugins-2g4c9","openshift-multus/multus-km6mf","openshift-network-diagnostics/network-check-target-bv96g","openshift-ovn-kubernetes/ovnkube-node-vfz5t"] Jul 16 17:33:44.280642 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.280623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.282875 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.282846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.282985 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.282952 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vj8sn\"" Jul 16 17:33:44.283060 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.282954 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:33:44.283118 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.283064 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 17:33:44.283168 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.283159 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 17:33:44.284607 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.284590 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 17:33:44.284961 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.284943 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 17:33:44.285063 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.284991 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 17:33:44.285063 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.285057 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.285236 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.285220 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 17:33:44.285333 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.285317 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 17:33:44.285589 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.285570 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mjnrc\"" Jul 16 17:33:44.287021 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.286996 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 17:33:44.287132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.287070 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 17:33:44.287477 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.287278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 17:33:44.287477 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.287295 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l84nq\"" Jul 16 17:33:44.287687 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.287671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.289339 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.289325 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 17:33:44.289572 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.289557 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 17:33:44.289670 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.289655 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfgd5\"" Jul 16 17:33:44.289736 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.289684 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 17:33:44.290077 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.290063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.291928 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.291910 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 17:33:44.292262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.292250 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h525p\"" Jul 16 17:33:44.292262 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.292257 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 17:33:44.292381 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.292310 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 17:33:44.292566 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.292553 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:44.292645 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.292624 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:44.292746 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.292731 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 17:33:44.292790 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.292754 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 17:33:44.292790 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.292765 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 17:33:44.294955 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.294897 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.296922 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.296905 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:33:44.297102 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.297083 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 17:33:44.297182 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.297105 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cnvkw\"" Jul 16 17:33:44.297345 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.297331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:44.297401 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.297386 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:44.298891 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.298872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-slash\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.298961 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.298895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-systemd\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.298961 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.298912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-etc-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.298961 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.298934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d34db377-5cb5-48a2-bf2d-24892a1cc619-iptables-alerter-script\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.298961 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.298958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrbn2\" (UniqueName: \"kubernetes.io/projected/d34db377-5cb5-48a2-bf2d-24892a1cc619-kube-api-access-zrbn2\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.299149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.298979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-env-overrides\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.299149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbzl\" (UniqueName: \"kubernetes.io/projected/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-kube-api-access-jwbzl\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.299149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-sys-fs\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.299149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b95008c8-7da1-4f94-93dc-4d871db2edc2-hosts-file\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.299149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-systemd-units\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjng\" (UniqueName: \"kubernetes.io/projected/28db96b1-887b-4387-b604-7eea68068a4b-kube-api-access-zxjng\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d34db377-5cb5-48a2-bf2d-24892a1cc619-host-slash\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr6m\" (UniqueName: \"kubernetes.io/projected/b95008c8-7da1-4f94-93dc-4d871db2edc2-kube-api-access-qkr6m\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-ovn\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-ovnkube-config\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28db96b1-887b-4387-b604-7eea68068a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.299676 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cnibin\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.299676 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-registration-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.299676 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.299789 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.299777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.300110 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-kubelet\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-var-lib-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300313 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-cni-bin\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300348 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-cni-netd\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300348 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-ovnkube-script-lib\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.300432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-node-log\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-device-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.300536 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.300536 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfvf\" (UniqueName: \"kubernetes.io/projected/6d26286c-330f-440b-bbda-8efe7a4f5304-kube-api-access-fcfvf\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.300536 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300477 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-run-netns\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300536 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-log-socket\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.300702 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-system-cni-dir\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.300702 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-os-release\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.300702 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-socket-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.300702 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.300609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95008c8-7da1-4f94-93dc-4d871db2edc2-tmp-dir\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.301932 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.301917 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 17:33:44.302006 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.301936 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 17:33:44.302169 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.302157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4vpbd\"" Jul 16 17:33:44.302237 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.302218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.303010 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.302995 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:33:44.306603 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.304530 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 17:33:44.306603 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.304526 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 17:33:44.306603 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.304661 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:33:44.306603 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.304715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r6mcv\"" Jul 16 17:33:44.306872 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.306846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.307243 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.307222 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:44.309686 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.309670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.311927 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.311904 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9vd4k\"" Jul 16 17:33:44.312001 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.311911 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 17:33:44.330869 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.330847 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8l5h9" Jul 16 17:33:44.338154 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.338137 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8l5h9" Jul 16 17:33:44.390652 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.390619 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 17:33:44.401016 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.400988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfvf\" (UniqueName: \"kubernetes.io/projected/6d26286c-330f-440b-bbda-8efe7a4f5304-kube-api-access-fcfvf\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.401174 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysctl-conf\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.401174 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-serviceca\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.401174 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-hostroot\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.401174 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-multus-certs\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.401174 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-run-netns\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-run-netns\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-os-release\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-cnibin\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95008c8-7da1-4f94-93dc-4d871db2edc2-tmp-dir\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401346 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-os-release\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-slash\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-systemd\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-etc-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401469 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-etc-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-slash\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d34db377-5cb5-48a2-bf2d-24892a1cc619-iptables-alerter-script\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cedc1c45-ce74-4f57-95ba-77eabe10b456-konnectivity-ca\") pod \"konnectivity-agent-rnkws\" (UID: \"cedc1c45-ce74-4f57-95ba-77eabe10b456\") " pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-systemd\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50cdddea-617e-4f65-acb6-927e443845ff-kubelet-config\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbzl\" (UniqueName: \"kubernetes.io/projected/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-kube-api-access-jwbzl\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-sys-fs\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95008c8-7da1-4f94-93dc-4d871db2edc2-tmp-dir\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-lib-modules\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-host\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-os-release\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-cni-multus\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.401805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-systemd-units\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjng\" (UniqueName: \"kubernetes.io/projected/28db96b1-887b-4387-b604-7eea68068a4b-kube-api-access-zxjng\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-sys-fs\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-modprobe-d\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqldv\" (UniqueName: \"kubernetes.io/projected/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-kube-api-access-vqldv\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f855705-abda-416d-b6a8-61b67eaf3684-cni-binary-copy\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hcw\" (UniqueName: \"kubernetes.io/projected/9f855705-abda-416d-b6a8-61b67eaf3684-kube-api-access-w7hcw\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-systemd-units\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.401991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28db96b1-887b-4387-b604-7eea68068a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-registration-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402052 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysctl-d\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-registration-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50cdddea-617e-4f65-acb6-927e443845ff-dbus\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d34db377-5cb5-48a2-bf2d-24892a1cc619-iptables-alerter-script\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bddb\" (UniqueName: \"kubernetes.io/projected/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-kube-api-access-8bddb\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-socket-dir-parent\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.402640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-host\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-var-lib-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-cni-netd\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-systemd\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-var-lib-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-openvswitch\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-k8s-cni-cncf-io\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402412 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-netns\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-etc-kubernetes\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-node-log\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-sys\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402544 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-node-log\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-log-socket\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-system-cni-dir\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-socket-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.403415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-cni-bin\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402690 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-system-cni-dir\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrbn2\" (UniqueName: \"kubernetes.io/projected/d34db377-5cb5-48a2-bf2d-24892a1cc619-kube-api-access-zrbn2\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-env-overrides\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-run\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-cni-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysconfig\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-var-lib-kubelet\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-socket-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b95008c8-7da1-4f94-93dc-4d871db2edc2-hosts-file\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.402989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d34db377-5cb5-48a2-bf2d-24892a1cc619-host-slash\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cedc1c45-ce74-4f57-95ba-77eabe10b456-agent-certs\") pod \"konnectivity-agent-rnkws\" (UID: \"cedc1c45-ce74-4f57-95ba-77eabe10b456\") " pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b95008c8-7da1-4f94-93dc-4d871db2edc2-hosts-file\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-cni-netd\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr6m\" (UniqueName: \"kubernetes.io/projected/b95008c8-7da1-4f94-93dc-4d871db2edc2-kube-api-access-qkr6m\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d34db377-5cb5-48a2-bf2d-24892a1cc619-host-slash\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-ovn\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-ovnkube-config\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-env-overrides\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cnibin\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-log-socket\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-system-cni-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-run-ovn\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-kubelet\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-tuned\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.404808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cnibin\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a556d059-3fe8-4189-9fa8-98c7000f55e0-tmp\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-conf-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f855705-abda-416d-b6a8-61b67eaf3684-multus-daemon-config\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-kubelet\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-kubelet\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-cni-bin\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-ovnkube-script-lib\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403850 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28db96b1-887b-4387-b604-7eea68068a4b-host-cni-bin\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-kubernetes\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tdf\" (UniqueName: \"kubernetes.io/projected/a556d059-3fe8-4189-9fa8-98c7000f55e0-kube-api-access-h7tdf\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-device-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.403956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-etc-selinux\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.404004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-device-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.405526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.404013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d26286c-330f-440b-bbda-8efe7a4f5304-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.406261 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.404013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.406261 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.404347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-ovnkube-config\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.406261 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.404352 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28db96b1-887b-4387-b604-7eea68068a4b-ovnkube-script-lib\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.407099 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.407075 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28db96b1-887b-4387-b604-7eea68068a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.412578 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.412520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfvf\" (UniqueName: \"kubernetes.io/projected/6d26286c-330f-440b-bbda-8efe7a4f5304-kube-api-access-fcfvf\") pod \"aws-ebs-csi-driver-node-x8nln\" (UID: \"6d26286c-330f-440b-bbda-8efe7a4f5304\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.412688 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.412661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbzl\" (UniqueName: \"kubernetes.io/projected/69eb91ad-a8ca-470a-9424-6c3d2dbb9320-kube-api-access-jwbzl\") pod \"multus-additional-cni-plugins-2g4c9\" (UID: \"69eb91ad-a8ca-470a-9424-6c3d2dbb9320\") " pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.413592 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.413569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjng\" (UniqueName: \"kubernetes.io/projected/28db96b1-887b-4387-b604-7eea68068a4b-kube-api-access-zxjng\") pod \"ovnkube-node-vfz5t\" (UID: \"28db96b1-887b-4387-b604-7eea68068a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.413749 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.413729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr6m\" (UniqueName: \"kubernetes.io/projected/b95008c8-7da1-4f94-93dc-4d871db2edc2-kube-api-access-qkr6m\") pod \"node-resolver-wkxtr\" (UID: \"b95008c8-7da1-4f94-93dc-4d871db2edc2\") " pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.414423 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.414408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrbn2\" (UniqueName: \"kubernetes.io/projected/d34db377-5cb5-48a2-bf2d-24892a1cc619-kube-api-access-zrbn2\") pod \"iptables-alerter-glk67\" (UID: \"d34db377-5cb5-48a2-bf2d-24892a1cc619\") " pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.504543 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-cni-bin\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-run\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-cni-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysconfig\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-cni-bin\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-var-lib-kubelet\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-run\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cedc1c45-ce74-4f57-95ba-77eabe10b456-agent-certs\") pod \"konnectivity-agent-rnkws\" (UID: \"cedc1c45-ce74-4f57-95ba-77eabe10b456\") " pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-var-lib-kubelet\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.504695 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:44.504717 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504721 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysconfig\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-system-cni-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-system-cni-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-kubelet\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.504781 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret podName:50cdddea-617e-4f65-acb6-927e443845ff nodeName:}" failed. No retries permitted until 2026-07-16 17:33:45.004754067 +0000 UTC m=+3.107857902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret") pod "global-pull-secret-syncer-kq8gb" (UID: "50cdddea-617e-4f65-acb6-927e443845ff") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-kubelet\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-tuned\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504860 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-cni-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a556d059-3fe8-4189-9fa8-98c7000f55e0-tmp\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-conf-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f855705-abda-416d-b6a8-61b67eaf3684-multus-daemon-config\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.504995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-kubernetes\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tdf\" (UniqueName: \"kubernetes.io/projected/a556d059-3fe8-4189-9fa8-98c7000f55e0-kube-api-access-h7tdf\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-conf-dir\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysctl-conf\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-serviceca\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-hostroot\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.505263 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.505102 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-multus-certs\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.505143 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:33:45.005128222 +0000 UTC m=+3.108232040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505155 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-multus-certs\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-cnibin\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cedc1c45-ce74-4f57-95ba-77eabe10b456-konnectivity-ca\") pod \"konnectivity-agent-rnkws\" (UID: \"cedc1c45-ce74-4f57-95ba-77eabe10b456\") " pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50cdddea-617e-4f65-acb6-927e443845ff-kubelet-config\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-lib-modules\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-host\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysctl-conf\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-os-release\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-os-release\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-cni-multus\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-modprobe-d\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqldv\" (UniqueName: \"kubernetes.io/projected/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-kube-api-access-vqldv\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f855705-abda-416d-b6a8-61b67eaf3684-cni-binary-copy\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506092 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hcw\" (UniqueName: \"kubernetes.io/projected/9f855705-abda-416d-b6a8-61b67eaf3684-kube-api-access-w7hcw\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysctl-d\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50cdddea-617e-4f65-acb6-927e443845ff-dbus\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bddb\" (UniqueName: \"kubernetes.io/projected/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-kube-api-access-8bddb\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505565 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-serviceca\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-socket-dir-parent\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-host\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-hostroot\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-systemd\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505679 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-cnibin\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50cdddea-617e-4f65-acb6-927e443845ff-kubelet-config\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-kubernetes\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505857 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f855705-abda-416d-b6a8-61b67eaf3684-multus-daemon-config\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.505999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-lib-modules\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-var-lib-cni-multus\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-modprobe-d\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cedc1c45-ce74-4f57-95ba-77eabe10b456-konnectivity-ca\") pod \"konnectivity-agent-rnkws\" (UID: \"cedc1c45-ce74-4f57-95ba-77eabe10b456\") " pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-host\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.506951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-sysctl-d\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-multus-socket-dir-parent\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50cdddea-617e-4f65-acb6-927e443845ff-dbus\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506474 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-host\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-systemd\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-k8s-cni-cncf-io\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-k8s-cni-cncf-io\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-netns\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-etc-kubernetes\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-host-run-netns\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-sys\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f855705-abda-416d-b6a8-61b67eaf3684-etc-kubernetes\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a556d059-3fe8-4189-9fa8-98c7000f55e0-sys\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.506880 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f855705-abda-416d-b6a8-61b67eaf3684-cni-binary-copy\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.507967 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a556d059-3fe8-4189-9fa8-98c7000f55e0-etc-tuned\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.508187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.507982 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cedc1c45-ce74-4f57-95ba-77eabe10b456-agent-certs\") pod \"konnectivity-agent-rnkws\" (UID: \"cedc1c45-ce74-4f57-95ba-77eabe10b456\") " pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.509043 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.509025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a556d059-3fe8-4189-9fa8-98c7000f55e0-tmp\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.512386 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.512366 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:33:44.512501 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.512393 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:33:44.512501 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.512414 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:33:44.512501 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.512426 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d55tb for pod openshift-network-diagnostics/network-check-target-bv96g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:44.512501 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:44.512479 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb podName:76540f10-29f6-4fe6-be47-760ea023e036 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:45.012465159 +0000 UTC m=+3.115569001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d55tb" (UniqueName: "kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb") pod "network-check-target-bv96g" (UID: "76540f10-29f6-4fe6-be47-760ea023e036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:44.515005 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.514976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hcw\" (UniqueName: \"kubernetes.io/projected/9f855705-abda-416d-b6a8-61b67eaf3684-kube-api-access-w7hcw\") pod \"multus-km6mf\" (UID: \"9f855705-abda-416d-b6a8-61b67eaf3684\") " pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.515110 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.515012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tdf\" (UniqueName: \"kubernetes.io/projected/a556d059-3fe8-4189-9fa8-98c7000f55e0-kube-api-access-h7tdf\") pod \"tuned-tb7tc\" (UID: \"a556d059-3fe8-4189-9fa8-98c7000f55e0\") " pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.515660 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.515528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bddb\" (UniqueName: \"kubernetes.io/projected/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-kube-api-access-8bddb\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:44.516800 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.516766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqldv\" (UniqueName: \"kubernetes.io/projected/a620fd44-37b6-4caa-8ecd-e2e785fff8bd-kube-api-access-vqldv\") pod \"node-ca-lfccv\" (UID: \"a620fd44-37b6-4caa-8ecd-e2e785fff8bd\") " pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.573707 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.573653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" event={"ID":"3af81e36783efc72a4cd6203a0dae2b4","Type":"ContainerStarted","Data":"6b4f8ddba1fb9317a42da990c82996529733c938785864a4a30eb1e7b04f436a"} Jul 16 17:33:44.574975 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.574913 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" event={"ID":"be2e4781f1bc5622abf5feee16446d86","Type":"ContainerStarted","Data":"22141d71ce7076bf8b1ff40c8e075e79614340c611c8831f96f0826746bc1be7"} Jul 16 17:33:44.591330 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.591296 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-glk67" Jul 16 17:33:44.597638 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.597614 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" Jul 16 17:33:44.604296 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.604262 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34db377_5cb5_48a2_bf2d_24892a1cc619.slice/crio-d5d98feacbfdd3c6094232c3423971faf9162659ddd80f1ed69ab63ebf540814 WatchSource:0}: Error finding container d5d98feacbfdd3c6094232c3423971faf9162659ddd80f1ed69ab63ebf540814: Status 404 returned error can't find the container with id d5d98feacbfdd3c6094232c3423971faf9162659ddd80f1ed69ab63ebf540814 Jul 16 17:33:44.607156 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.607137 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" Jul 16 17:33:44.609979 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.609920 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69eb91ad_a8ca_470a_9424_6c3d2dbb9320.slice/crio-5a8734f6a6d298576f95e620e18ad29ff98a133134b3bf3b71aafe599da3f876 WatchSource:0}: Error finding container 5a8734f6a6d298576f95e620e18ad29ff98a133134b3bf3b71aafe599da3f876: Status 404 returned error can't find the container with id 5a8734f6a6d298576f95e620e18ad29ff98a133134b3bf3b71aafe599da3f876 Jul 16 17:33:44.612833 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.612814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wkxtr" Jul 16 17:33:44.616209 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.616184 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d26286c_330f_440b_bbda_8efe7a4f5304.slice/crio-2dd5b718e831b8dd97b4435b25ddd6f1a82f5ded0ad22d4fc5e68332eb4bdf3f WatchSource:0}: Error finding container 2dd5b718e831b8dd97b4435b25ddd6f1a82f5ded0ad22d4fc5e68332eb4bdf3f: Status 404 returned error can't find the container with id 2dd5b718e831b8dd97b4435b25ddd6f1a82f5ded0ad22d4fc5e68332eb4bdf3f Jul 16 17:33:44.617867 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.617849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:33:44.623980 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.623959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" Jul 16 17:33:44.626272 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.626251 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95008c8_7da1_4f94_93dc_4d871db2edc2.slice/crio-d18d49e0790ff3cc379848b848e089b5080bbe5962c877033678118d52637302 WatchSource:0}: Error finding container d18d49e0790ff3cc379848b848e089b5080bbe5962c877033678118d52637302: Status 404 returned error can't find the container with id d18d49e0790ff3cc379848b848e089b5080bbe5962c877033678118d52637302 Jul 16 17:33:44.628289 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.628271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:33:44.633488 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.633469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lfccv" Jul 16 17:33:44.635580 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.635556 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda556d059_3fe8_4189_9fa8_98c7000f55e0.slice/crio-5347dacfc0e6d9f6d99d21a27539aae41b03622bf6fe8f83acfe1da500bf06d7 WatchSource:0}: Error finding container 5347dacfc0e6d9f6d99d21a27539aae41b03622bf6fe8f83acfe1da500bf06d7: Status 404 returned error can't find the container with id 5347dacfc0e6d9f6d99d21a27539aae41b03622bf6fe8f83acfe1da500bf06d7 Jul 16 17:33:44.637888 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:44.637875 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-km6mf" Jul 16 17:33:44.640862 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.640840 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedc1c45_ce74_4f57_95ba_77eabe10b456.slice/crio-028a11adc424cd0b7357850691a41900a9402e8204f97a69f1a07c0b3f1d9966 WatchSource:0}: Error finding container 028a11adc424cd0b7357850691a41900a9402e8204f97a69f1a07c0b3f1d9966: Status 404 returned error can't find the container with id 028a11adc424cd0b7357850691a41900a9402e8204f97a69f1a07c0b3f1d9966 Jul 16 17:33:44.645366 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.645342 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda620fd44_37b6_4caa_8ecd_e2e785fff8bd.slice/crio-c18e0d9429cae8f99b09320d946f1e4431b7300fad188254618d6ebb2596abe9 WatchSource:0}: Error finding container c18e0d9429cae8f99b09320d946f1e4431b7300fad188254618d6ebb2596abe9: Status 404 returned error can't find the container with id c18e0d9429cae8f99b09320d946f1e4431b7300fad188254618d6ebb2596abe9 Jul 16 17:33:44.650942 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:33:44.650904 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f855705_abda_416d_b6a8_61b67eaf3684.slice/crio-9cd93ed1bf0368b555523e4b7216d7f237c3af72f9bfe92713822fd3cd52382a WatchSource:0}: Error finding container 9cd93ed1bf0368b555523e4b7216d7f237c3af72f9bfe92713822fd3cd52382a: Status 404 returned error can't find the container with id 9cd93ed1bf0368b555523e4b7216d7f237c3af72f9bfe92713822fd3cd52382a Jul 16 17:33:45.010558 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.010463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:45.010558 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.010533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:45.010777 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.010629 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:45.010777 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.010681 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret podName:50cdddea-617e-4f65-acb6-927e443845ff nodeName:}" failed. No retries permitted until 2026-07-16 17:33:46.010666461 +0000 UTC m=+4.113770279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret") pod "global-pull-secret-syncer-kq8gb" (UID: "50cdddea-617e-4f65-acb6-927e443845ff") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:45.010777 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.010629 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:45.010777 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.010758 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:33:46.010746148 +0000 UTC m=+4.113849985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:45.111689 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.111659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:45.111866 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.111799 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:33:45.111866 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.111823 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:33:45.111866 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.111835 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d55tb for pod openshift-network-diagnostics/network-check-target-bv96g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:45.111994 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.111919 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb podName:76540f10-29f6-4fe6-be47-760ea023e036 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:46.11188399 +0000 UTC m=+4.214987809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d55tb" (UniqueName: "kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb") pod "network-check-target-bv96g" (UID: "76540f10-29f6-4fe6-be47-760ea023e036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:45.341429 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.341368 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:28:45 +0000 UTC" deadline="2027-12-29 11:46:51.516690146 +0000 UTC" Jul 16 17:33:45.341429 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.341412 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12738h13m6.175283014s" Jul 16 17:33:45.505809 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.505775 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:33:45.568771 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.568734 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:45.568965 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.568863 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:45.568965 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.568934 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:45.569088 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:45.569038 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:45.579992 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.579760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-km6mf" event={"ID":"9f855705-abda-416d-b6a8-61b67eaf3684","Type":"ContainerStarted","Data":"9cd93ed1bf0368b555523e4b7216d7f237c3af72f9bfe92713822fd3cd52382a"} Jul 16 17:33:45.581322 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.581248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lfccv" event={"ID":"a620fd44-37b6-4caa-8ecd-e2e785fff8bd","Type":"ContainerStarted","Data":"c18e0d9429cae8f99b09320d946f1e4431b7300fad188254618d6ebb2596abe9"} Jul 16 17:33:45.583127 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.583099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rnkws" event={"ID":"cedc1c45-ce74-4f57-95ba-77eabe10b456","Type":"ContainerStarted","Data":"028a11adc424cd0b7357850691a41900a9402e8204f97a69f1a07c0b3f1d9966"} Jul 16 17:33:45.585683 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.585610 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"26430d2a7b0e598bc1af619f55573bee333b269ada3ff744b1418d8ab49997ef"} Jul 16 17:33:45.588826 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.587318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" event={"ID":"6d26286c-330f-440b-bbda-8efe7a4f5304","Type":"ContainerStarted","Data":"2dd5b718e831b8dd97b4435b25ddd6f1a82f5ded0ad22d4fc5e68332eb4bdf3f"} Jul 16 17:33:45.588957 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.588825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerStarted","Data":"5a8734f6a6d298576f95e620e18ad29ff98a133134b3bf3b71aafe599da3f876"} Jul 16 17:33:45.590414 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.590300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-glk67" event={"ID":"d34db377-5cb5-48a2-bf2d-24892a1cc619","Type":"ContainerStarted","Data":"d5d98feacbfdd3c6094232c3423971faf9162659ddd80f1ed69ab63ebf540814"} Jul 16 17:33:45.591970 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.591897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" event={"ID":"a556d059-3fe8-4189-9fa8-98c7000f55e0","Type":"ContainerStarted","Data":"5347dacfc0e6d9f6d99d21a27539aae41b03622bf6fe8f83acfe1da500bf06d7"} Jul 16 17:33:45.593760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:45.593734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wkxtr" event={"ID":"b95008c8-7da1-4f94-93dc-4d871db2edc2","Type":"ContainerStarted","Data":"d18d49e0790ff3cc379848b848e089b5080bbe5962c877033678118d52637302"} Jul 16 17:33:46.021080 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:46.021036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:46.021286 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:46.021145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:46.021357 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.021302 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:46.021410 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.021373 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret podName:50cdddea-617e-4f65-acb6-927e443845ff nodeName:}" failed. No retries permitted until 2026-07-16 17:33:48.021351287 +0000 UTC m=+6.124455155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret") pod "global-pull-secret-syncer-kq8gb" (UID: "50cdddea-617e-4f65-acb6-927e443845ff") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:46.021781 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.021681 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:46.021781 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.021754 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:33:48.021734141 +0000 UTC m=+6.124837987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:46.122151 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:46.122055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:46.122341 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.122310 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:33:46.122402 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.122348 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:33:46.122402 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.122364 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d55tb for pod openshift-network-diagnostics/network-check-target-bv96g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:46.122525 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.122431 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb podName:76540f10-29f6-4fe6-be47-760ea023e036 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:48.122412603 +0000 UTC m=+6.225516447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d55tb" (UniqueName: "kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb") pod "network-check-target-bv96g" (UID: "76540f10-29f6-4fe6-be47-760ea023e036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:46.342244 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:46.342176 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:28:45 +0000 UTC" deadline="2028-01-12 06:14:48.313785503 +0000 UTC" Jul 16 17:33:46.342244 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:46.342242 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13068h41m1.971547588s" Jul 16 17:33:46.571301 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:46.571275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:46.571493 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:46.571403 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:46.602595 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:46.602558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" event={"ID":"be2e4781f1bc5622abf5feee16446d86","Type":"ContainerStarted","Data":"ddaf0215418f8db18c33e0ffd4ba0fe33e3474368bc9e3aa3f55721db34105bf"} Jul 16 17:33:47.568907 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:47.568834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:47.569308 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:47.569030 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:47.569666 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:47.569479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:47.569666 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:47.569608 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:48.040658 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:48.040619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:48.040856 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:48.040691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:48.040964 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.040906 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:48.041034 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.040971 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:33:52.040951597 +0000 UTC m=+10.144055427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:48.041395 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.041375 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:48.041479 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.041467 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret podName:50cdddea-617e-4f65-acb6-927e443845ff nodeName:}" failed. No retries permitted until 2026-07-16 17:33:52.041448916 +0000 UTC m=+10.144552742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret") pod "global-pull-secret-syncer-kq8gb" (UID: "50cdddea-617e-4f65-acb6-927e443845ff") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:48.141865 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:48.141514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:48.141865 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.141701 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:33:48.141865 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.141729 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:33:48.141865 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.141743 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d55tb for pod openshift-network-diagnostics/network-check-target-bv96g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:48.141865 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.141802 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb podName:76540f10-29f6-4fe6-be47-760ea023e036 nodeName:}" failed. No retries permitted until 2026-07-16 17:33:52.141782468 +0000 UTC m=+10.244886310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d55tb" (UniqueName: "kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb") pod "network-check-target-bv96g" (UID: "76540f10-29f6-4fe6-be47-760ea023e036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:48.571987 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:48.571463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:48.571987 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:48.571591 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:49.569414 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:49.569379 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:49.569590 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:49.569529 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:49.569959 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:49.569937 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:49.570081 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:49.570044 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:50.569226 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:50.568484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:50.569226 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:50.568608 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:51.569394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:51.569352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:51.569931 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:51.569518 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:51.569931 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:51.569692 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:51.569931 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:51.569793 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:52.077380 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:52.077336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:52.077599 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:52.077405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:52.077599 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.077527 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:52.077599 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.077587 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:34:00.077566344 +0000 UTC m=+18.180670168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:33:52.077976 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.077956 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:52.078085 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.078001 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret podName:50cdddea-617e-4f65-acb6-927e443845ff nodeName:}" failed. No retries permitted until 2026-07-16 17:34:00.07799007 +0000 UTC m=+18.181093890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret") pod "global-pull-secret-syncer-kq8gb" (UID: "50cdddea-617e-4f65-acb6-927e443845ff") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:33:52.178549 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:52.178508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:52.178718 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.178686 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:33:52.178718 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.178708 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:33:52.178845 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.178723 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d55tb for pod openshift-network-diagnostics/network-check-target-bv96g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:52.178845 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.178791 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb podName:76540f10-29f6-4fe6-be47-760ea023e036 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:00.178772316 +0000 UTC m=+18.281876151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d55tb" (UniqueName: "kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb") pod "network-check-target-bv96g" (UID: "76540f10-29f6-4fe6-be47-760ea023e036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:33:52.570480 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:52.570022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:52.570480 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:52.570149 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:53.569317 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:53.569277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:53.569519 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:53.569279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:53.569519 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:53.569422 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:53.569630 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:53.569511 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:54.569472 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:54.569444 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:54.569955 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:54.569581 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:55.568800 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:55.568765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:55.568985 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:55.568765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:55.568985 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:55.568892 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:55.568985 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:55.568948 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:56.568820 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:56.568792 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:56.569192 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:56.568920 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:57.569286 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:57.569247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:57.569764 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:57.569247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:57.569764 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:57.569399 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:33:57.569764 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:57.569483 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:58.571273 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:58.571243 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:33:58.571652 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:58.571362 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:33:59.568818 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:59.568779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:33:59.568818 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:33:59.568798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:33:59.569090 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:59.568919 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:33:59.569090 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:33:59.569061 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:00.144859 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:00.144802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:00.145439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:00.144875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:00.145439 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.144965 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:00.145439 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.144966 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:00.145439 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.145027 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:34:16.145011195 +0000 UTC m=+34.248115019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:00.145439 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.145048 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret podName:50cdddea-617e-4f65-acb6-927e443845ff nodeName:}" failed. No retries permitted until 2026-07-16 17:34:16.145037331 +0000 UTC m=+34.248141151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret") pod "global-pull-secret-syncer-kq8gb" (UID: "50cdddea-617e-4f65-acb6-927e443845ff") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:00.245891 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:00.245857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:00.246073 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.246048 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:00.246073 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.246071 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:00.246171 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.246083 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d55tb for pod openshift-network-diagnostics/network-check-target-bv96g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:00.246171 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.246141 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb podName:76540f10-29f6-4fe6-be47-760ea023e036 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:16.246126516 +0000 UTC m=+34.349230338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d55tb" (UniqueName: "kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb") pod "network-check-target-bv96g" (UID: "76540f10-29f6-4fe6-be47-760ea023e036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:00.571562 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:00.571529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:00.571741 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:00.571643 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:01.568740 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:01.568707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:01.569194 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:01.568743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:01.569194 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:01.568876 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:01.569194 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:01.569014 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:02.569588 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:02.569540 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:02.570056 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:02.569650 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:03.569303 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:03.569264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:03.569483 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:03.569279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:03.569483 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:03.569408 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:03.569483 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:03.569444 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:04.568962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:04.568924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:04.569408 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:04.569060 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:05.568803 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:05.568782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:05.568917 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:05.568898 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:05.568966 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:05.568950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:05.569064 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:05.569048 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:05.649038 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:05.648974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" event={"ID":"a556d059-3fe8-4189-9fa8-98c7000f55e0","Type":"ContainerStarted","Data":"7f8074ba863798df729c72b4c80cf7890d17c5075ab26a753f7ca085049387bd"} Jul 16 17:34:05.669405 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:05.669349 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-249.ec2.internal" podStartSLOduration=21.669335127 podStartE2EDuration="21.669335127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:33:46.624180471 +0000 UTC m=+4.727284316" watchObservedRunningTime="2026-07-16 17:34:05.669335127 +0000 UTC m=+23.772438977" Jul 16 17:34:06.569426 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.569076 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:06.570090 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:06.569471 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:06.652291 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.652262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" event={"ID":"6d26286c-330f-440b-bbda-8efe7a4f5304","Type":"ContainerStarted","Data":"b5a609bcf4f0dae7d8ffc5b35a32fba926dab4dc96a0d8ef189b23b67d0e883a"} Jul 16 17:34:06.653609 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.653587 2564 generic.go:356] "Generic (PLEG): container finished" podID="69eb91ad-a8ca-470a-9424-6c3d2dbb9320" containerID="ef32a99411dcb80ae977b3737922d34aaa10fe1c34116e24d125468ecfe5787b" exitCode=0 Jul 16 17:34:06.653687 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.653671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerDied","Data":"ef32a99411dcb80ae977b3737922d34aaa10fe1c34116e24d125468ecfe5787b"} Jul 16 17:34:06.656533 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.656515 2564 generic.go:356] "Generic (PLEG): container finished" podID="3af81e36783efc72a4cd6203a0dae2b4" containerID="811b03d19efcfbefb19405da8b0329fa36587e6f4f91677c88e7b10bd5303b04" exitCode=0 Jul 16 17:34:06.656596 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.656584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" event={"ID":"3af81e36783efc72a4cd6203a0dae2b4","Type":"ContainerDied","Data":"811b03d19efcfbefb19405da8b0329fa36587e6f4f91677c88e7b10bd5303b04"} Jul 16 17:34:06.658148 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.657994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wkxtr" event={"ID":"b95008c8-7da1-4f94-93dc-4d871db2edc2","Type":"ContainerStarted","Data":"7f596e5863f6ff44338007c8333799124c02664531d463e99e039024dad2c5e7"} Jul 16 17:34:06.659481 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.659462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-km6mf" event={"ID":"9f855705-abda-416d-b6a8-61b67eaf3684","Type":"ContainerStarted","Data":"19fd8f34d507ce11dd3f084a9ac2f9114329256d40337dd639195a38da7cc125"} Jul 16 17:34:06.660850 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.660815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lfccv" event={"ID":"a620fd44-37b6-4caa-8ecd-e2e785fff8bd","Type":"ContainerStarted","Data":"01db3a5fe22811b96a4761afbc79b6d0c4ea9c55212e324fa7a55e4b80878cf2"} Jul 16 17:34:06.662028 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.662009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rnkws" event={"ID":"cedc1c45-ce74-4f57-95ba-77eabe10b456","Type":"ContainerStarted","Data":"08fdf0e7cdf5664fd7f6a1ed1e9c7776ab28480f8b83a273073db37a6720f5b6"} Jul 16 17:34:06.664346 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664332 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:34:06.664620 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664601 2564 generic.go:356] "Generic (PLEG): container finished" podID="28db96b1-887b-4387-b604-7eea68068a4b" containerID="855466927e9145029265ca74f13fe5e8b09af41a910a4680d5d3061c89842e30" exitCode=1 Jul 16 17:34:06.664680 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664657 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"e0cdfa887c9cb0de13cde540ff5510ec5eb4058b649453f0a94c1c33c99ad339"} Jul 16 17:34:06.664715 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"eb8b108f1be0aa60129f77402e672c540fde2843ad82cab4191158d8e3c69d7f"} Jul 16 17:34:06.664752 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"92a3d93e92b45b9bb9c6965410112e90c03a87d27b8e77a4f9ee03d8f73eec78"} Jul 16 17:34:06.664792 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664772 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"a6635344abcf947b759d3869ebe926ffa52f4bbed146e430f19a612a424823bd"} Jul 16 17:34:06.664849 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerDied","Data":"855466927e9145029265ca74f13fe5e8b09af41a910a4680d5d3061c89842e30"} Jul 16 17:34:06.664901 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.664847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"1e4e483d42f6d2762a231c536c759e6d308629953b0e4148aba95b91d658bcda"} Jul 16 17:34:06.683194 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.683144 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tb7tc" podStartSLOduration=2.852479867 podStartE2EDuration="23.683130485s" totalImagesPullingTime="20.830650618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:05.669586034 +0000 UTC m=+23.772689853" watchObservedRunningTime="2026-07-16 17:34:06.683130485 +0000 UTC m=+24.786234325" Jul 16 17:34:06.701712 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.701662 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rnkws" podStartSLOduration=2.874932054 podStartE2EDuration="23.701648807s" totalImagesPullingTime="20.826716753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:06.700720317 +0000 UTC m=+24.803824157" watchObservedRunningTime="2026-07-16 17:34:06.701648807 +0000 UTC m=+24.804752646" Jul 16 17:34:06.730962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.730916 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lfccv" podStartSLOduration=7.791250389 podStartE2EDuration="23.730900393s" totalImagesPullingTime="15.939650004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:06.730334792 +0000 UTC m=+24.833438632" watchObservedRunningTime="2026-07-16 17:34:06.730900393 +0000 UTC m=+24.834004270" Jul 16 17:34:06.745919 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.745880 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wkxtr" podStartSLOduration=3.016927013 podStartE2EDuration="23.745865367s" totalImagesPullingTime="20.728938354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:06.745498393 +0000 UTC m=+24.848602232" watchObservedRunningTime="2026-07-16 17:34:06.745865367 +0000 UTC m=+24.848969203" Jul 16 17:34:06.764507 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:06.764465 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-km6mf" podStartSLOduration=2.941284379 podStartE2EDuration="23.764450512s" totalImagesPullingTime="20.823166133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:06.764186853 +0000 UTC m=+24.867290692" watchObservedRunningTime="2026-07-16 17:34:06.764450512 +0000 UTC m=+24.867554353" Jul 16 17:34:07.279404 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:07.279373 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:34:07.568757 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:07.568677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:07.568999 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:07.568777 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:07.568999 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:07.568846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:07.568999 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:07.568931 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:07.668151 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:07.668118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-glk67" event={"ID":"d34db377-5cb5-48a2-bf2d-24892a1cc619","Type":"ContainerStarted","Data":"399af7abe6e79ef214f4f719222a960a1a9c752c43d46072208dc2098b865ce1"} Jul 16 17:34:07.669983 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:07.669960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" event={"ID":"3af81e36783efc72a4cd6203a0dae2b4","Type":"ContainerStarted","Data":"03d62b346fb9180e1aef9c0eb0d68ce843aac226857a58ec4bb393d34c2ab7d0"} Jul 16 17:34:07.689105 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:07.689069 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-glk67" podStartSLOduration=3.826705227 podStartE2EDuration="24.689057093s" totalImagesPullingTime="20.862351866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:07.688570804 +0000 UTC m=+25.791674643" watchObservedRunningTime="2026-07-16 17:34:07.689057093 +0000 UTC m=+25.792160932" Jul 16 17:34:07.707777 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:07.707744 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-249.ec2.internal" podStartSLOduration=23.707732098 podStartE2EDuration="23.707732098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:07.705077888 +0000 UTC m=+25.808181727" watchObservedRunningTime="2026-07-16 17:34:07.707732098 +0000 UTC m=+25.810835938" Jul 16 17:34:08.132431 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:08.132393 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:34:08.132954 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:08.132935 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:34:08.569451 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:08.569431 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:08.569579 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:08.569520 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:08.673093 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:08.672954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rnkws" Jul 16 17:34:08.734031 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:08.733997 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 17:34:09.371541 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.371442 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:34:08.73401483Z","UUID":"c3dcf5d0-2aec-429a-be63-b46a92355b29","Handler":null,"Name":"","Endpoint":""} Jul 16 17:34:09.373106 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.373086 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 17:34:09.373218 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.373138 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 17:34:09.569353 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.569320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:09.569518 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.569364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:09.569518 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:09.569476 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:09.569621 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:09.569587 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:09.678819 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.678757 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:34:09.679353 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.679206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"9b754ef52e3cc9c827d75f099108718039fd73493d97f3a4f7e5fa32ce9122e4"} Jul 16 17:34:09.681261 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:09.681219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" event={"ID":"6d26286c-330f-440b-bbda-8efe7a4f5304","Type":"ContainerStarted","Data":"5fb74214ebee886f7583eae18447dc4a0af6078b9a3c2b5b8d27a44dca6b56d3"} Jul 16 17:34:10.568718 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:10.568681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:10.568909 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:10.568865 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:11.568880 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:11.568842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:11.568880 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:11.568873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:11.569523 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:11.568965 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:11.569523 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:11.569065 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:12.570121 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:12.569905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:12.570509 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:12.570220 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:12.692802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:12.692780 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:34:12.693163 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:12.693138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"aa1d2df6ebd5fd22ce7816ffad255d13afbd81ac9181de2dc96f893bc40c9169"} Jul 16 17:34:12.693414 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:12.693391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:34:12.693649 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:12.693629 2564 scope.go:117] "RemoveContainer" containerID="855466927e9145029265ca74f13fe5e8b09af41a910a4680d5d3061c89842e30" Jul 16 17:34:12.695175 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:12.695145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerStarted","Data":"81ecbbccbc46479b3ac081e630d32a975d77909c4e1d7a9dd99b9a07abbe13aa"} Jul 16 17:34:12.708858 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:12.708813 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:34:13.569015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.568968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:13.569015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.569005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:13.569379 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:13.569098 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:13.569379 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:13.569222 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:13.700908 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.700880 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:34:13.701362 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.701246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" event={"ID":"28db96b1-887b-4387-b604-7eea68068a4b","Type":"ContainerStarted","Data":"02305d54614b0d7797e7b2128755156122e8c255617beda4afce8e3a6a2100cb"} Jul 16 17:34:13.701433 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.701412 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:34:13.701491 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.701467 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:34:13.703472 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.703447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" event={"ID":"6d26286c-330f-440b-bbda-8efe7a4f5304","Type":"ContainerStarted","Data":"5c72ba8b37e74c9c673d6534bf47ec402d3d4396010bafbafe888cdc8f878a32"} Jul 16 17:34:13.705244 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.705222 2564 generic.go:356] "Generic (PLEG): container finished" podID="69eb91ad-a8ca-470a-9424-6c3d2dbb9320" containerID="81ecbbccbc46479b3ac081e630d32a975d77909c4e1d7a9dd99b9a07abbe13aa" exitCode=0 Jul 16 17:34:13.705339 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.705262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerDied","Data":"81ecbbccbc46479b3ac081e630d32a975d77909c4e1d7a9dd99b9a07abbe13aa"} Jul 16 17:34:13.716820 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.716800 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:34:13.728737 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.728688 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" podStartSLOduration=9.558948283 podStartE2EDuration="30.728672737s" totalImagesPullingTime="21.169724454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:13.728245719 +0000 UTC m=+31.831349584" watchObservedRunningTime="2026-07-16 17:34:13.728672737 +0000 UTC m=+31.831776579" Jul 16 17:34:13.803740 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:13.803690 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x8nln" podStartSLOduration=3.241612769 podStartE2EDuration="30.80367456s" totalImagesPullingTime="27.562061791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:13.776514619 +0000 UTC m=+31.879618459" watchObservedRunningTime="2026-07-16 17:34:13.80367456 +0000 UTC m=+31.906778400" Jul 16 17:34:14.569372 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:14.569179 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:14.569518 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:14.569492 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:14.786980 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:14.786950 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87b84"] Jul 16 17:34:14.787379 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:14.787073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:14.787379 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:14.787195 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:14.788198 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:14.788169 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bv96g"] Jul 16 17:34:14.788321 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:14.788268 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:14.788384 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:14.788351 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:14.789487 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:14.789459 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kq8gb"] Jul 16 17:34:14.789620 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:14.789595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:14.789727 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:14.789705 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:16.161666 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:16.161634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:16.162139 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:16.161691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:16.162139 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.161772 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:16.162139 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.161794 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:16.162139 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.161848 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:34:48.161820608 +0000 UTC m=+66.264924427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:34:16.162139 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.161863 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret podName:50cdddea-617e-4f65-acb6-927e443845ff nodeName:}" failed. No retries permitted until 2026-07-16 17:34:48.161856811 +0000 UTC m=+66.264960632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret") pod "global-pull-secret-syncer-kq8gb" (UID: "50cdddea-617e-4f65-acb6-927e443845ff") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:34:16.263029 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:16.262990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:16.263187 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.263142 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:34:16.263187 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.263162 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:34:16.263187 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.263181 2564 projected.go:196] Error preparing data for projected volume kube-api-access-d55tb for pod openshift-network-diagnostics/network-check-target-bv96g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:16.263288 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.263227 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb podName:76540f10-29f6-4fe6-be47-760ea023e036 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:48.263214 +0000 UTC m=+66.366317821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d55tb" (UniqueName: "kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb") pod "network-check-target-bv96g" (UID: "76540f10-29f6-4fe6-be47-760ea023e036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:34:16.568892 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:16.568849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:16.569077 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:16.568853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:16.569077 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.568999 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b84" podUID="3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc" Jul 16 17:34:16.569077 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:16.568853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:16.569077 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.569065 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bv96g" podUID="76540f10-29f6-4fe6-be47-760ea023e036" Jul 16 17:34:16.569274 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:16.569117 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kq8gb" podUID="50cdddea-617e-4f65-acb6-927e443845ff" Jul 16 17:34:18.209070 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.209043 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-249.ec2.internal" event="NodeReady" Jul 16 17:34:18.209548 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.209172 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 17:34:18.270860 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.270607 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z54xl"] Jul 16 17:34:18.277517 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.277492 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-22r9v"] Jul 16 17:34:18.277645 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.277626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.280183 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.280156 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ntp5m\"" Jul 16 17:34:18.280289 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.280211 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 17:34:18.280458 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.280440 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 17:34:18.282474 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.281795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.282574 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.282547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-22r9v"] Jul 16 17:34:18.284003 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.283988 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 17:34:18.286368 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.284769 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 17:34:18.286368 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.284884 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 17:34:18.286368 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.285189 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n568h\"" Jul 16 17:34:18.291281 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.291244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z54xl"] Jul 16 17:34:18.380765 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.380738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fb5d45c-4ace-43e5-9420-744cb4d952d2-cert\") pod \"ingress-canary-22r9v\" (UID: \"0fb5d45c-4ace-43e5-9420-744cb4d952d2\") " pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.380881 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.380806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.380944 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.380883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41658552-fcc5-420f-80c1-9078dfb115a6-config-volume\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.381013 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.380994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41658552-fcc5-420f-80c1-9078dfb115a6-tmp-dir\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.381054 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.381041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hqn\" (UniqueName: \"kubernetes.io/projected/0fb5d45c-4ace-43e5-9420-744cb4d952d2-kube-api-access-52hqn\") pod \"ingress-canary-22r9v\" (UID: \"0fb5d45c-4ace-43e5-9420-744cb4d952d2\") " pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.381093 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.381062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jr67\" (UniqueName: \"kubernetes.io/projected/41658552-fcc5-420f-80c1-9078dfb115a6-kube-api-access-9jr67\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.482125 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41658552-fcc5-420f-80c1-9078dfb115a6-config-volume\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.482125 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41658552-fcc5-420f-80c1-9078dfb115a6-tmp-dir\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.482351 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hqn\" (UniqueName: \"kubernetes.io/projected/0fb5d45c-4ace-43e5-9420-744cb4d952d2-kube-api-access-52hqn\") pod \"ingress-canary-22r9v\" (UID: \"0fb5d45c-4ace-43e5-9420-744cb4d952d2\") " pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.482351 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jr67\" (UniqueName: \"kubernetes.io/projected/41658552-fcc5-420f-80c1-9078dfb115a6-kube-api-access-9jr67\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.482351 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fb5d45c-4ace-43e5-9420-744cb4d952d2-cert\") pod \"ingress-canary-22r9v\" (UID: \"0fb5d45c-4ace-43e5-9420-744cb4d952d2\") " pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.482351 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.482351 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:18.482290 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:18.482549 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:18.482363 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:18.982345195 +0000 UTC m=+37.085449013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:34:18.482597 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482543 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41658552-fcc5-420f-80c1-9078dfb115a6-tmp-dir\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.482679 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.482655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41658552-fcc5-420f-80c1-9078dfb115a6-config-volume\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.486453 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.486431 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fb5d45c-4ace-43e5-9420-744cb4d952d2-cert\") pod \"ingress-canary-22r9v\" (UID: \"0fb5d45c-4ace-43e5-9420-744cb4d952d2\") " pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.498404 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.498375 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jr67\" (UniqueName: \"kubernetes.io/projected/41658552-fcc5-420f-80c1-9078dfb115a6-kube-api-access-9jr67\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.498608 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.498587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hqn\" (UniqueName: \"kubernetes.io/projected/0fb5d45c-4ace-43e5-9420-744cb4d952d2-kube-api-access-52hqn\") pod \"ingress-canary-22r9v\" (UID: \"0fb5d45c-4ace-43e5-9420-744cb4d952d2\") " pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.568564 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.568521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:18.568746 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.568525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:18.568746 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.568667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:18.571150 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.571129 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:34:18.571296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.571179 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:34:18.571296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.571248 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9hcv\"" Jul 16 17:34:18.571296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.571282 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:34:18.571520 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.571498 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:34:18.571587 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.571533 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s9dx5\"" Jul 16 17:34:18.614715 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.614689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-22r9v" Jul 16 17:34:18.718909 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.718683 2564 generic.go:356] "Generic (PLEG): container finished" podID="69eb91ad-a8ca-470a-9424-6c3d2dbb9320" containerID="cd76fb5c6852c8fd2e4d373f01314635581d4db727fa877759a6601c9af4d3ac" exitCode=0 Jul 16 17:34:18.719028 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.718755 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerDied","Data":"cd76fb5c6852c8fd2e4d373f01314635581d4db727fa877759a6601c9af4d3ac"} Jul 16 17:34:18.793650 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.793619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-22r9v"] Jul 16 17:34:18.797306 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:34:18.797280 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb5d45c_4ace_43e5_9420_744cb4d952d2.slice/crio-a618c44583493ad12b31d9d992ad1ed37fddda4e76685a21222440aec940e924 WatchSource:0}: Error finding container a618c44583493ad12b31d9d992ad1ed37fddda4e76685a21222440aec940e924: Status 404 returned error can't find the container with id a618c44583493ad12b31d9d992ad1ed37fddda4e76685a21222440aec940e924 Jul 16 17:34:18.986068 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:18.986032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:18.986226 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:18.986154 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:18.986226 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:18.986212 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:19.986195231 +0000 UTC m=+38.089299069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:34:19.722319 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:19.722286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-22r9v" event={"ID":"0fb5d45c-4ace-43e5-9420-744cb4d952d2","Type":"ContainerStarted","Data":"a618c44583493ad12b31d9d992ad1ed37fddda4e76685a21222440aec940e924"} Jul 16 17:34:19.992147 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:19.992061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:19.992287 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:19.992210 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:19.992287 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:19.992286 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:21.992269501 +0000 UTC m=+40.095373323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:34:20.726999 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:20.726968 2564 generic.go:356] "Generic (PLEG): container finished" podID="69eb91ad-a8ca-470a-9424-6c3d2dbb9320" containerID="f4058331a2f5ec853c6eecf416256d7b2ab6e32f0968f78193c531b51f060dbd" exitCode=0 Jul 16 17:34:20.727446 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:20.727004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerDied","Data":"f4058331a2f5ec853c6eecf416256d7b2ab6e32f0968f78193c531b51f060dbd"} Jul 16 17:34:22.006743 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:22.006606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:22.007267 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:22.006790 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:22.007267 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:22.006893 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:26.006870327 +0000 UTC m=+44.109974162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:34:23.736998 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:23.736912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-22r9v" event={"ID":"0fb5d45c-4ace-43e5-9420-744cb4d952d2","Type":"ContainerStarted","Data":"9943f2d13f29115708119587ea7f87042d3b3a997ce5b360c007183731098f69"} Jul 16 17:34:23.753723 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:23.753669 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-22r9v" podStartSLOduration=1.119355317 podStartE2EDuration="5.7536482s" totalImagesPullingTime="4.634292883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:23.7529959 +0000 UTC m=+41.856099741" watchObservedRunningTime="2026-07-16 17:34:23.7536482 +0000 UTC m=+41.856752022" Jul 16 17:34:25.744779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:25.744692 2564 generic.go:356] "Generic (PLEG): container finished" podID="69eb91ad-a8ca-470a-9424-6c3d2dbb9320" containerID="e399b813dd82da089b1a9c6d969b78dbbdd1686a2db7ee4ddf3a65905d85cf29" exitCode=0 Jul 16 17:34:25.744779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:25.744740 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerDied","Data":"e399b813dd82da089b1a9c6d969b78dbbdd1686a2db7ee4ddf3a65905d85cf29"} Jul 16 17:34:26.036498 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:26.036469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:26.036648 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:26.036606 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:26.036685 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:26.036669 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:34.036654289 +0000 UTC m=+52.139758110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:34:26.749618 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:26.749580 2564 generic.go:356] "Generic (PLEG): container finished" podID="69eb91ad-a8ca-470a-9424-6c3d2dbb9320" containerID="15ac355cffc6d262261457c6f00b14f40e68dececf60fffcbed8506bacf590be" exitCode=0 Jul 16 17:34:26.749971 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:26.749636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerDied","Data":"15ac355cffc6d262261457c6f00b14f40e68dececf60fffcbed8506bacf590be"} Jul 16 17:34:27.754874 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:27.754838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" event={"ID":"69eb91ad-a8ca-470a-9424-6c3d2dbb9320","Type":"ContainerStarted","Data":"f07f44fb5bcf5019719c252cf7281e42c082ff8e200d3a1e06646f4163191cac"} Jul 16 17:34:27.786102 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:27.786047 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2g4c9" podStartSLOduration=7.355584548 podStartE2EDuration="44.786031778s" totalImagesPullingTime="37.43044723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:27.78447219 +0000 UTC m=+45.887576030" watchObservedRunningTime="2026-07-16 17:34:27.786031778 +0000 UTC m=+45.889135617" Jul 16 17:34:34.087506 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:34.087460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:34.087985 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:34.087598 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:34.087985 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:34.087649 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:50.087635507 +0000 UTC m=+68.190739325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:34:45.719659 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:45.719633 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfz5t" Jul 16 17:34:48.179781 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.179720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:48.179781 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.179786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:34:48.182252 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.182228 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:34:48.182312 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.182236 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:34:48.190187 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:48.190163 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 17:34:48.190307 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:48.190232 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs podName:3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc nodeName:}" failed. No retries permitted until 2026-07-16 17:35:52.190211674 +0000 UTC m=+130.293315492 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs") pod "network-metrics-daemon-87b84" (UID: "3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc") : secret "metrics-daemon-secret" not found Jul 16 17:34:48.193989 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.193965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50cdddea-617e-4f65-acb6-927e443845ff-original-pull-secret\") pod \"global-pull-secret-syncer-kq8gb\" (UID: \"50cdddea-617e-4f65-acb6-927e443845ff\") " pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:48.280118 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.280095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:48.282596 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.282575 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:34:48.289225 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.289200 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kq8gb" Jul 16 17:34:48.292199 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.292186 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:34:48.303794 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.303770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d55tb\" (UniqueName: \"kubernetes.io/projected/76540f10-29f6-4fe6-be47-760ea023e036-kube-api-access-d55tb\") pod \"network-check-target-bv96g\" (UID: \"76540f10-29f6-4fe6-be47-760ea023e036\") " pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:48.408213 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.408184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kq8gb"] Jul 16 17:34:48.413570 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:34:48.413541 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cdddea_617e_4f65_acb6_927e443845ff.slice/crio-35569764b8571ea854f4eb0b753e1d6375746ce7b8418a656cc104743a2b74f0 WatchSource:0}: Error finding container 35569764b8571ea854f4eb0b753e1d6375746ce7b8418a656cc104743a2b74f0: Status 404 returned error can't find the container with id 35569764b8571ea854f4eb0b753e1d6375746ce7b8418a656cc104743a2b74f0 Jul 16 17:34:48.581306 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.581277 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s9dx5\"" Jul 16 17:34:48.588988 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.588969 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:48.714007 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.713975 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bv96g"] Jul 16 17:34:48.716938 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:34:48.716909 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76540f10_29f6_4fe6_be47_760ea023e036.slice/crio-d9f6dd3c26574edc09a376f88a053176af83d4ef4b7ea173c96092e726827f86 WatchSource:0}: Error finding container d9f6dd3c26574edc09a376f88a053176af83d4ef4b7ea173c96092e726827f86: Status 404 returned error can't find the container with id d9f6dd3c26574edc09a376f88a053176af83d4ef4b7ea173c96092e726827f86 Jul 16 17:34:48.803213 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.803179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bv96g" event={"ID":"76540f10-29f6-4fe6-be47-760ea023e036","Type":"ContainerStarted","Data":"d9f6dd3c26574edc09a376f88a053176af83d4ef4b7ea173c96092e726827f86"} Jul 16 17:34:48.804178 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:48.804159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kq8gb" event={"ID":"50cdddea-617e-4f65-acb6-927e443845ff","Type":"ContainerStarted","Data":"35569764b8571ea854f4eb0b753e1d6375746ce7b8418a656cc104743a2b74f0"} Jul 16 17:34:50.092694 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:50.092647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:34:50.093088 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:50.092780 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:34:50.093088 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:34:50.092860 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.092844258 +0000 UTC m=+100.195948077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:34:53.818747 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:53.818715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bv96g" event={"ID":"76540f10-29f6-4fe6-be47-760ea023e036","Type":"ContainerStarted","Data":"ecd0df12647d8a7c6ef9136b1422428f87944ca1016255bc8f3690c5894d32b8"} Jul 16 17:34:53.819173 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:53.818848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:34:53.820292 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:53.820261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kq8gb" event={"ID":"50cdddea-617e-4f65-acb6-927e443845ff","Type":"ContainerStarted","Data":"7357d32ebdfe4f3a914cc7c7c9547118669d60f764b81b729d21934be9f2c00a"} Jul 16 17:34:53.836121 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:53.836077 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bv96g" podStartSLOduration=66.491359469 podStartE2EDuration="1m10.836061516s" totalImagesPullingTime="4.344702047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:53.835542876 +0000 UTC m=+71.938646715" watchObservedRunningTime="2026-07-16 17:34:53.836061516 +0000 UTC m=+71.939165355" Jul 16 17:34:53.851233 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:34:53.851191 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kq8gb" podStartSLOduration=66.200052663 podStartE2EDuration="1m10.851178422s" totalImagesPullingTime="4.651125759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:53.851042509 +0000 UTC m=+71.954146348" watchObservedRunningTime="2026-07-16 17:34:53.851178422 +0000 UTC m=+71.954282260" Jul 16 17:35:15.546809 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.546665 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td"] Jul 16 17:35:15.549719 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.549695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" Jul 16 17:35:15.550228 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.550204 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-dhckd"] Jul 16 17:35:15.551909 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.551891 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.552090 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.552068 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mmc7v\"" Jul 16 17:35:15.557380 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.557361 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 17:35:15.558186 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.558169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 16 17:35:15.558358 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.558335 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 17:35:15.560244 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.560228 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 16 17:35:15.560333 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.560230 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8dplq\"" Jul 16 17:35:15.563240 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.563221 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td"] Jul 16 17:35:15.566370 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.566348 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 16 17:35:15.570237 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.570213 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-dhckd"] Jul 16 17:35:15.654177 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.654136 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78d54cf4db-8dh9v"] Jul 16 17:35:15.655943 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.655928 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77cb777ccb-d74d9"] Jul 16 17:35:15.656061 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.656045 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.657669 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.657646 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.658767 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.658747 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 16 17:35:15.658767 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.658758 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 16 17:35:15.658947 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.658814 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 16 17:35:15.658947 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.658758 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 17:35:15.659199 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.659182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 17:35:15.659366 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.659350 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hzm9b\"" Jul 16 17:35:15.659473 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.659459 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 16 17:35:15.659990 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.659975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gcp44\"" Jul 16 17:35:15.660596 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.660577 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 17:35:15.661251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.661230 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 17:35:15.661251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.661238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 17:35:15.666257 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-snapshots\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.666257 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.666404 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhfv\" (UniqueName: \"kubernetes.io/projected/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-kube-api-access-qmhfv\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.666404 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-serving-cert\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.666502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqp8\" (UniqueName: \"kubernetes.io/projected/acce4b1f-453f-41ff-939b-240d335957e3-kube-api-access-jhqp8\") pod \"network-check-source-869dcfcdcd-kc2td\" (UID: \"acce4b1f-453f-41ff-939b-240d335957e3\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" Jul 16 17:35:15.666502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.666595 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-tmp\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.666666 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.666651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 17:35:15.674806 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.674787 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78d54cf4db-8dh9v"] Jul 16 17:35:15.677644 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.677617 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77cb777ccb-d74d9"] Jul 16 17:35:15.767020 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.766984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-tmp\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.767020 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.767238 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767238 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-stats-auth\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.767238 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767102 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-installation-pull-secrets\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767238 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-bound-sa-token\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767238 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-snapshots\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhfv\" (UniqueName: \"kubernetes.io/projected/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-kube-api-access-qmhfv\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqlz\" (UniqueName: \"kubernetes.io/projected/86262674-0c27-49a1-89a3-01071d6fbdaf-kube-api-access-fvqlz\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-image-registry-private-configuration\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-tmp\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c79l\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-kube-api-access-2c79l\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-serving-cert\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.767444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98415fd2-f4bd-4028-9841-f50b879fef4f-ca-trust-extracted\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.767760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-certificates\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-trusted-ca\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.767760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqp8\" (UniqueName: \"kubernetes.io/projected/acce4b1f-453f-41ff-939b-240d335957e3-kube-api-access-jhqp8\") pod \"network-check-source-869dcfcdcd-kc2td\" (UID: \"acce4b1f-453f-41ff-939b-240d335957e3\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" Jul 16 17:35:15.767760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.767760 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-default-certificate\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.767987 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.767970 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-snapshots\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.768110 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.768093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.768222 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.768201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.769417 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.769400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-serving-cert\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.781333 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.781304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhfv\" (UniqueName: \"kubernetes.io/projected/80e75ef9-a0a6-4037-8db0-3cd0c08369c6-kube-api-access-qmhfv\") pod \"insights-operator-68d9dd94cf-dhckd\" (UID: \"80e75ef9-a0a6-4037-8db0-3cd0c08369c6\") " pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.781748 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.781730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqp8\" (UniqueName: \"kubernetes.io/projected/acce4b1f-453f-41ff-939b-240d335957e3-kube-api-access-jhqp8\") pod \"network-check-source-869dcfcdcd-kc2td\" (UID: \"acce4b1f-453f-41ff-939b-240d335957e3\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" Jul 16 17:35:15.858931 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.858819 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" Jul 16 17:35:15.865735 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.865710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" Jul 16 17:35:15.868936 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.868914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-default-certificate\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.869054 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.868959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.869054 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.868978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.869054 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.869008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-stats-auth\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.869054 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.869032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-installation-pull-secrets\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.869252 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:15.869059 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:35:15.869252 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:15.869112 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.369094652 +0000 UTC m=+94.472198483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : secret "router-metrics-certs-default" not found Jul 16 17:35:15.869252 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.869058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-bound-sa-token\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.869396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.869258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqlz\" (UniqueName: \"kubernetes.io/projected/86262674-0c27-49a1-89a3-01071d6fbdaf-kube-api-access-fvqlz\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.869396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.869282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-image-registry-private-configuration\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.869396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.869305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c79l\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-kube-api-access-2c79l\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.870212 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.870187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98415fd2-f4bd-4028-9841-f50b879fef4f-ca-trust-extracted\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.870272 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.870260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.870905 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.870881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-certificates\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.871096 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:15.871073 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.371051851 +0000 UTC m=+94.474155685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : configmap references non-existent config key: service-ca.crt Jul 16 17:35:15.871648 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:15.871196 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:35:15.871648 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:15.871212 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77cb777ccb-d74d9: secret "image-registry-tls" not found Jul 16 17:35:15.871648 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:15.871259 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls podName:98415fd2-f4bd-4028-9841-f50b879fef4f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:16.371244807 +0000 UTC m=+94.474348638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls") pod "image-registry-77cb777ccb-d74d9" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f") : secret "image-registry-tls" not found Jul 16 17:35:15.872774 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.872753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-default-certificate\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.874412 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.874383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98415fd2-f4bd-4028-9841-f50b879fef4f-ca-trust-extracted\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.878449 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.874699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-certificates\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.878449 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.874750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-trusted-ca\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.878449 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.876426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-stats-auth\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.878449 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.878308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-bound-sa-token\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.879469 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.879389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-image-registry-private-configuration\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.879529 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.879486 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-installation-pull-secrets\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.880015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.879993 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-trusted-ca\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.888032 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.888004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqlz\" (UniqueName: \"kubernetes.io/projected/86262674-0c27-49a1-89a3-01071d6fbdaf-kube-api-access-fvqlz\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:15.889104 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.889082 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c79l\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-kube-api-access-2c79l\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:15.991874 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:15.991805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td"] Jul 16 17:35:15.995385 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:15.995361 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacce4b1f_453f_41ff_939b_240d335957e3.slice/crio-2a1f1aadde51f89e3c3a6d306a26bea4ebff769c8145f79d1e07887da2aebfde WatchSource:0}: Error finding container 2a1f1aadde51f89e3c3a6d306a26bea4ebff769c8145f79d1e07887da2aebfde: Status 404 returned error can't find the container with id 2a1f1aadde51f89e3c3a6d306a26bea4ebff769c8145f79d1e07887da2aebfde Jul 16 17:35:16.004116 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.004094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-dhckd"] Jul 16 17:35:16.006973 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:16.006948 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e75ef9_a0a6_4037_8db0_3cd0c08369c6.slice/crio-4ebe29d6c8bc53ee5bd868380a77dae523f0a3e31896cfb4116c951a322eeeab WatchSource:0}: Error finding container 4ebe29d6c8bc53ee5bd868380a77dae523f0a3e31896cfb4116c951a322eeeab: Status 404 returned error can't find the container with id 4ebe29d6c8bc53ee5bd868380a77dae523f0a3e31896cfb4116c951a322eeeab Jul 16 17:35:16.379979 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.379941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:16.379979 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.379980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:16.380191 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.380025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:16.380191 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:16.380110 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:35:16.380191 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:16.380151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:17.380138052 +0000 UTC m=+95.483241870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : configmap references non-existent config key: service-ca.crt Jul 16 17:35:16.380191 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:16.380158 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:35:16.380191 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:16.380174 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77cb777ccb-d74d9: secret "image-registry-tls" not found Jul 16 17:35:16.380191 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:16.380177 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:17.380159265 +0000 UTC m=+95.483263097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : secret "router-metrics-certs-default" not found Jul 16 17:35:16.380373 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:16.380225 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls podName:98415fd2-f4bd-4028-9841-f50b879fef4f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:17.380214699 +0000 UTC m=+95.483318516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls") pod "image-registry-77cb777ccb-d74d9" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f") : secret "image-registry-tls" not found Jul 16 17:35:16.880736 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.880692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" event={"ID":"80e75ef9-a0a6-4037-8db0-3cd0c08369c6","Type":"ContainerStarted","Data":"4ebe29d6c8bc53ee5bd868380a77dae523f0a3e31896cfb4116c951a322eeeab"} Jul 16 17:35:16.882555 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.882527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" event={"ID":"acce4b1f-453f-41ff-939b-240d335957e3","Type":"ContainerStarted","Data":"cfabfcc68549026b224bd50793b56b053cb9b0f361594efbfb3f88b1fb16f1c3"} Jul 16 17:35:16.882683 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.882572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" event={"ID":"acce4b1f-453f-41ff-939b-240d335957e3","Type":"ContainerStarted","Data":"2a1f1aadde51f89e3c3a6d306a26bea4ebff769c8145f79d1e07887da2aebfde"} Jul 16 17:35:16.900136 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:16.900078 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-kc2td" podStartSLOduration=1.90006014 podStartE2EDuration="1.90006014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:16.89913339 +0000 UTC m=+95.002237239" watchObservedRunningTime="2026-07-16 17:35:16.90006014 +0000 UTC m=+95.003163980" Jul 16 17:35:17.388350 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:17.388312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:17.388566 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:17.388401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:17.388566 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:17.388427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:17.388566 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:17.388503 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.38848051 +0000 UTC m=+97.491584330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : configmap references non-existent config key: service-ca.crt Jul 16 17:35:17.388566 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:17.388511 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:35:17.388566 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:17.388546 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.388539141 +0000 UTC m=+97.491642958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : secret "router-metrics-certs-default" not found Jul 16 17:35:17.388803 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:17.388572 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:35:17.388803 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:17.388591 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77cb777ccb-d74d9: secret "image-registry-tls" not found Jul 16 17:35:17.388803 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:17.388648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls podName:98415fd2-f4bd-4028-9841-f50b879fef4f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.388631685 +0000 UTC m=+97.491735522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls") pod "image-registry-77cb777ccb-d74d9" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f") : secret "image-registry-tls" not found Jul 16 17:35:17.888218 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:17.888173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" event={"ID":"80e75ef9-a0a6-4037-8db0-3cd0c08369c6","Type":"ContainerStarted","Data":"8075c4a99eaf6916756847cd7ddfde32949b44edca02d17c0bdfa1401609e596"} Jul 16 17:35:17.961536 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:17.961486 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" podStartSLOduration=1.2406009980000001 podStartE2EDuration="2.961469471s" totalImagesPullingTime="1.720868473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:17.947623267 +0000 UTC m=+96.050727103" watchObservedRunningTime="2026-07-16 17:35:17.961469471 +0000 UTC m=+96.064573308" Jul 16 17:35:19.019691 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.019661 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gfv6q"] Jul 16 17:35:19.021676 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.021651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.023902 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.023877 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 17:35:19.024047 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.023906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lbz87\"" Jul 16 17:35:19.024145 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.024134 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 17:35:19.036440 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.036315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gfv6q"] Jul 16 17:35:19.102906 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.102879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98e95ef5-11f0-459f-8722-0d796d9b78d3-crio-socket\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.103034 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.102926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e95ef5-11f0-459f-8722-0d796d9b78d3-data-volume\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.103034 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.102988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6p9b\" (UniqueName: \"kubernetes.io/projected/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-api-access-d6p9b\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.103106 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.103034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.103106 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.103085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204285 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.204251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98e95ef5-11f0-459f-8722-0d796d9b78d3-crio-socket\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204438 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.204297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e95ef5-11f0-459f-8722-0d796d9b78d3-data-volume\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204438 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.204372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98e95ef5-11f0-459f-8722-0d796d9b78d3-crio-socket\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204438 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.204417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6p9b\" (UniqueName: \"kubernetes.io/projected/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-api-access-d6p9b\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204529 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.204474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204563 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.204543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204596 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.204560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98e95ef5-11f0-459f-8722-0d796d9b78d3-data-volume\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.204640 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.204623 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:19.204671 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.204642 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:19.204698 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.204685 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.704666343 +0000 UTC m=+97.807770171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : configmap "kube-rbac-proxy" not found Jul 16 17:35:19.204736 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.204702 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.704693169 +0000 UTC m=+97.807796986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:19.215977 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.215948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6p9b\" (UniqueName: \"kubernetes.io/projected/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-api-access-d6p9b\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.406487 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.406451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:19.406659 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.406508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:19.406659 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.406603 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:35:19.406659 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.406627 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77cb777ccb-d74d9: secret "image-registry-tls" not found Jul 16 17:35:19.406759 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.406609 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.406595925 +0000 UTC m=+101.509699742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : configmap references non-existent config key: service-ca.crt Jul 16 17:35:19.406759 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.406682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:19.406851 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.406754 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:35:19.406851 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.406767 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls podName:98415fd2-f4bd-4028-9841-f50b879fef4f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.406752019 +0000 UTC m=+101.509855837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls") pod "image-registry-77cb777ccb-d74d9" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f") : secret "image-registry-tls" not found Jul 16 17:35:19.406851 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.406797 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.406786554 +0000 UTC m=+101.509890375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : secret "router-metrics-certs-default" not found Jul 16 17:35:19.709451 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.709361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.709451 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:19.709451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:19.709667 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.709522 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:19.709667 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.709554 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:19.709667 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.709569 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:20.709556524 +0000 UTC m=+98.812660341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : configmap "kube-rbac-proxy" not found Jul 16 17:35:19.709823 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:19.709684 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:20.709667158 +0000 UTC m=+98.812770991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:20.715685 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:20.715647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:20.716210 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:20.715745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:20.716210 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:20.715805 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:20.716210 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:20.715890 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.715871553 +0000 UTC m=+100.818975371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:20.716210 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:20.715903 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:20.716210 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:20.715969 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.715950293 +0000 UTC m=+100.819054116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : configmap "kube-rbac-proxy" not found Jul 16 17:35:21.190450 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:21.190424 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wkxtr_b95008c8-7da1-4f94-93dc-4d871db2edc2/dns-node-resolver/0.log" Jul 16 17:35:22.127375 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:22.127320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:35:22.127774 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:22.127475 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 17:35:22.127774 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:22.127538 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls podName:41658552-fcc5-420f-80c1-9078dfb115a6 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:26.127522939 +0000 UTC m=+164.230626756 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls") pod "dns-default-z54xl" (UID: "41658552-fcc5-420f-80c1-9078dfb115a6") : secret "dns-default-metrics-tls" not found Jul 16 17:35:22.188775 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:22.188745 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lfccv_a620fd44-37b6-4caa-8ecd-e2e785fff8bd/node-ca/0.log" Jul 16 17:35:22.588621 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:22.588596 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-22r9v_0fb5d45c-4ace-43e5-9420-744cb4d952d2/serve-healthcheck-canary/0.log" Jul 16 17:35:22.731807 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:22.731783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:22.731950 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:22.731893 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:22.731950 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:22.731938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:22.732035 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:22.731947 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:26.73193466 +0000 UTC m=+104.835038481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : configmap "kube-rbac-proxy" not found Jul 16 17:35:22.732079 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:22.732046 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:22.732121 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:22.732086 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:26.732075947 +0000 UTC m=+104.835179765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:23.438260 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:23.438227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:23.438282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:23.438305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:23.438374 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:23.438385 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:23.438393 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77cb777ccb-d74d9: secret "image-registry-tls" not found Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:23.438435 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:31.438422999 +0000 UTC m=+109.541526816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : secret "router-metrics-certs-default" not found Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:23.438454 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:31.438440165 +0000 UTC m=+109.541543983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : configmap references non-existent config key: service-ca.crt Jul 16 17:35:23.438674 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:23.438474 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls podName:98415fd2-f4bd-4028-9841-f50b879fef4f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:31.438467568 +0000 UTC m=+109.541571385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls") pod "image-registry-77cb777ccb-d74d9" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f") : secret "image-registry-tls" not found Jul 16 17:35:24.824766 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:24.824733 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bv96g" Jul 16 17:35:26.765289 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:26.765249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:26.765709 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:26.765320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:26.765709 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:26.765370 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:35:26.765709 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:26.765418 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:26.765709 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:26.765438 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:34.765421889 +0000 UTC m=+112.868525706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : configmap "kube-rbac-proxy" not found Jul 16 17:35:26.765709 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:26.765468 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:34.765456633 +0000 UTC m=+112.868560450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:31.505646 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:31.505611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:31.505682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:31.505719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:31.505758 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:31.505839 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:47.50580932 +0000 UTC m=+125.608913138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : secret "router-metrics-certs-default" not found Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:31.505848 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:31.505866 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77cb777ccb-d74d9: secret "image-registry-tls" not found Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:31.505858 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle podName:86262674-0c27-49a1-89a3-01071d6fbdaf nodeName:}" failed. No retries permitted until 2026-07-16 17:35:47.505851241 +0000 UTC m=+125.608955058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle") pod "router-default-78d54cf4db-8dh9v" (UID: "86262674-0c27-49a1-89a3-01071d6fbdaf") : configmap references non-existent config key: service-ca.crt Jul 16 17:35:31.506024 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:31.505931 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls podName:98415fd2-f4bd-4028-9841-f50b879fef4f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:47.505906528 +0000 UTC m=+125.609010351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls") pod "image-registry-77cb777ccb-d74d9" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f") : secret "image-registry-tls" not found Jul 16 17:35:34.832753 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:34.832717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:34.833145 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:34.832794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:34.833145 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:34.832927 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:35:34.833145 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:34.832986 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls podName:98e95ef5-11f0-459f-8722-0d796d9b78d3 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:50.832967644 +0000 UTC m=+128.936071481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gfv6q" (UID: "98e95ef5-11f0-459f-8722-0d796d9b78d3") : secret "insights-runtime-extractor-tls" not found Jul 16 17:35:34.833253 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:34.833197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98e95ef5-11f0-459f-8722-0d796d9b78d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:47.534074 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.534035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:47.534074 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.534081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:47.534587 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.534140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:47.534632 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.534595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86262674-0c27-49a1-89a3-01071d6fbdaf-service-ca-bundle\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:47.536420 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.536398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"image-registry-77cb777ccb-d74d9\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:47.536516 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.536426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86262674-0c27-49a1-89a3-01071d6fbdaf-metrics-certs\") pod \"router-default-78d54cf4db-8dh9v\" (UID: \"86262674-0c27-49a1-89a3-01071d6fbdaf\") " pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:47.768120 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.768082 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hzm9b\"" Jul 16 17:35:47.772551 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.772531 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gcp44\"" Jul 16 17:35:47.776295 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.776282 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:47.781372 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.781350 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:47.910179 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.910147 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78d54cf4db-8dh9v"] Jul 16 17:35:47.912852 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:47.912814 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86262674_0c27_49a1_89a3_01071d6fbdaf.slice/crio-6ea48d24d49d02f3262f01c3796e52b21c697614b2d693ccf38f91f6d937b4bd WatchSource:0}: Error finding container 6ea48d24d49d02f3262f01c3796e52b21c697614b2d693ccf38f91f6d937b4bd: Status 404 returned error can't find the container with id 6ea48d24d49d02f3262f01c3796e52b21c697614b2d693ccf38f91f6d937b4bd Jul 16 17:35:47.920463 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.920437 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77cb777ccb-d74d9"] Jul 16 17:35:47.923313 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:47.923285 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98415fd2_f4bd_4028_9841_f50b879fef4f.slice/crio-4f393cdc9f2796141eb5d1188cb2c0ae869a0b47fdffe4ac9640cb364244d4b9 WatchSource:0}: Error finding container 4f393cdc9f2796141eb5d1188cb2c0ae869a0b47fdffe4ac9640cb364244d4b9: Status 404 returned error can't find the container with id 4f393cdc9f2796141eb5d1188cb2c0ae869a0b47fdffe4ac9640cb364244d4b9 Jul 16 17:35:47.976879 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.976845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" event={"ID":"86262674-0c27-49a1-89a3-01071d6fbdaf","Type":"ContainerStarted","Data":"34b936c0ee6722e3341305cefa5fbde8b5383a5dc9063b2ac6dc6a5a973deb9e"} Jul 16 17:35:47.977028 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.976974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" event={"ID":"86262674-0c27-49a1-89a3-01071d6fbdaf","Type":"ContainerStarted","Data":"6ea48d24d49d02f3262f01c3796e52b21c697614b2d693ccf38f91f6d937b4bd"} Jul 16 17:35:47.978434 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.978408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" event={"ID":"98415fd2-f4bd-4028-9841-f50b879fef4f","Type":"ContainerStarted","Data":"83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317"} Jul 16 17:35:47.978549 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.978451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" event={"ID":"98415fd2-f4bd-4028-9841-f50b879fef4f","Type":"ContainerStarted","Data":"4f393cdc9f2796141eb5d1188cb2c0ae869a0b47fdffe4ac9640cb364244d4b9"} Jul 16 17:35:47.978549 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.978527 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:35:47.996496 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:47.996450 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" podStartSLOduration=32.9964354 podStartE2EDuration="32.9964354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:47.995624736 +0000 UTC m=+126.098728588" watchObservedRunningTime="2026-07-16 17:35:47.9964354 +0000 UTC m=+126.099539238" Jul 16 17:35:48.018605 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:48.018558 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" podStartSLOduration=33.018542685 podStartE2EDuration="33.018542685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:48.017171834 +0000 UTC m=+126.120275666" watchObservedRunningTime="2026-07-16 17:35:48.018542685 +0000 UTC m=+126.121646549" Jul 16 17:35:48.777472 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:48.777440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:48.780094 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:48.780073 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:48.981440 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:48.981408 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:48.982522 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:48.982501 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78d54cf4db-8dh9v" Jul 16 17:35:50.861687 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:50.861653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:50.863834 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:50.863810 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98e95ef5-11f0-459f-8722-0d796d9b78d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gfv6q\" (UID: \"98e95ef5-11f0-459f-8722-0d796d9b78d3\") " pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:51.134081 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.134013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lbz87\"" Jul 16 17:35:51.142884 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.142868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gfv6q" Jul 16 17:35:51.269911 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.269882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gfv6q"] Jul 16 17:35:51.272654 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:51.272631 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e95ef5_11f0_459f_8722_0d796d9b78d3.slice/crio-e829ba464366629d62fb8be59afef32914709a950f30b3ad05bbf687acf2ebfb WatchSource:0}: Error finding container e829ba464366629d62fb8be59afef32914709a950f30b3ad05bbf687acf2ebfb: Status 404 returned error can't find the container with id e829ba464366629d62fb8be59afef32914709a950f30b3ad05bbf687acf2ebfb Jul 16 17:35:51.386076 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.386008 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77cb777ccb-d74d9"] Jul 16 17:35:51.493512 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.493481 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t"] Jul 16 17:35:51.497957 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.497942 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" Jul 16 17:35:51.503740 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.503709 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r2nfq\"" Jul 16 17:35:51.503870 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.503758 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 17:35:51.508378 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.508358 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-tj4nn"] Jul 16 17:35:51.511354 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.511337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-tj4nn" Jul 16 17:35:51.534310 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.534288 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 17:35:51.534560 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.534546 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 17:35:51.537928 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.537905 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vr4wz\"" Jul 16 17:35:51.538466 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.538449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t"] Jul 16 17:35:51.539257 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.539235 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d4887f7bd-bktzp"] Jul 16 17:35:51.542397 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.542381 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.547280 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.547261 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 17:35:51.547388 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.547261 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 17:35:51.547388 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.547303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fb559\"" Jul 16 17:35:51.567430 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-oauth-config\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.567565 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-oauth-serving-cert\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.567565 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f2d0e39-8681-4c1c-9cf2-739835451b6b-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-5c85t\" (UID: \"1f2d0e39-8681-4c1c-9cf2-739835451b6b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" Jul 16 17:35:51.567565 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plpfg\" (UniqueName: \"kubernetes.io/projected/04b6d468-7be6-43da-ba89-88cde195a8f3-kube-api-access-plpfg\") pod \"downloads-68c9cbb497-tj4nn\" (UID: \"04b6d468-7be6-43da-ba89-88cde195a8f3\") " pod="openshift-console/downloads-68c9cbb497-tj4nn" Jul 16 17:35:51.567565 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-console-config\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.567565 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dk9\" (UniqueName: \"kubernetes.io/projected/4989298b-df06-474c-8ee6-f65524c2987a-kube-api-access-b6dk9\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.567742 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-serving-cert\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.567742 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.567602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-service-ca\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.568643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.568629 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 17:35:51.568709 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.568681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 17:35:51.568757 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.568744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 17:35:51.569688 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.569673 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6565964545-dhkr2"] Jul 16 17:35:51.572770 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.572755 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-tj4nn"] Jul 16 17:35:51.572860 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.572848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.640326 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.640248 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4887f7bd-bktzp"] Jul 16 17:35:51.648350 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.648319 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6565964545-dhkr2"] Jul 16 17:35:51.668084 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-console-config\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.668229 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dk9\" (UniqueName: \"kubernetes.io/projected/4989298b-df06-474c-8ee6-f65524c2987a-kube-api-access-b6dk9\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.668229 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa1250f-3079-4f3e-a981-515e8ec72ae8-registry-certificates\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668229 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-serving-cert\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.668229 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-service-ca\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.668229 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-bound-sa-token\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668478 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1250f-3079-4f3e-a981-515e8ec72ae8-trusted-ca\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668478 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpxw\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-kube-api-access-nqpxw\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668478 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-registry-tls\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668478 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa1250f-3079-4f3e-a981-515e8ec72ae8-image-registry-private-configuration\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668478 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa1250f-3079-4f3e-a981-515e8ec72ae8-installation-pull-secrets\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668478 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-oauth-config\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.668763 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-oauth-serving-cert\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.668763 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f2d0e39-8681-4c1c-9cf2-739835451b6b-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-5c85t\" (UID: \"1f2d0e39-8681-4c1c-9cf2-739835451b6b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" Jul 16 17:35:51.668763 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa1250f-3079-4f3e-a981-515e8ec72ae8-ca-trust-extracted\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.668763 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpfg\" (UniqueName: \"kubernetes.io/projected/04b6d468-7be6-43da-ba89-88cde195a8f3-kube-api-access-plpfg\") pod \"downloads-68c9cbb497-tj4nn\" (UID: \"04b6d468-7be6-43da-ba89-88cde195a8f3\") " pod="openshift-console/downloads-68c9cbb497-tj4nn" Jul 16 17:35:51.669031 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-console-config\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.669031 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.668958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-service-ca\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.669121 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.669103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-oauth-serving-cert\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.671063 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.671044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-serving-cert\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.671168 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.671149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f2d0e39-8681-4c1c-9cf2-739835451b6b-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-5c85t\" (UID: \"1f2d0e39-8681-4c1c-9cf2-739835451b6b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" Jul 16 17:35:51.671222 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.671203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-oauth-config\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.679750 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.679729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dk9\" (UniqueName: \"kubernetes.io/projected/4989298b-df06-474c-8ee6-f65524c2987a-kube-api-access-b6dk9\") pod \"console-5d4887f7bd-bktzp\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.696311 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.696292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plpfg\" (UniqueName: \"kubernetes.io/projected/04b6d468-7be6-43da-ba89-88cde195a8f3-kube-api-access-plpfg\") pod \"downloads-68c9cbb497-tj4nn\" (UID: \"04b6d468-7be6-43da-ba89-88cde195a8f3\") " pod="openshift-console/downloads-68c9cbb497-tj4nn" Jul 16 17:35:51.769090 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa1250f-3079-4f3e-a981-515e8ec72ae8-ca-trust-extracted\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769249 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa1250f-3079-4f3e-a981-515e8ec72ae8-registry-certificates\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769249 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-bound-sa-token\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769249 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1250f-3079-4f3e-a981-515e8ec72ae8-trusted-ca\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769249 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpxw\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-kube-api-access-nqpxw\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769437 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-registry-tls\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769437 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa1250f-3079-4f3e-a981-515e8ec72ae8-image-registry-private-configuration\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769437 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa1250f-3079-4f3e-a981-515e8ec72ae8-installation-pull-secrets\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.769578 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.769507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fa1250f-3079-4f3e-a981-515e8ec72ae8-ca-trust-extracted\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.770031 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.770007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fa1250f-3079-4f3e-a981-515e8ec72ae8-registry-certificates\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.770147 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.770115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1250f-3079-4f3e-a981-515e8ec72ae8-trusted-ca\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.771610 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.771586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fa1250f-3079-4f3e-a981-515e8ec72ae8-installation-pull-secrets\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.771713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.771697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-registry-tls\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.771751 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.771715 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fa1250f-3079-4f3e-a981-515e8ec72ae8-image-registry-private-configuration\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.781546 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.781530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-bound-sa-token\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.789714 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.789698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpxw\" (UniqueName: \"kubernetes.io/projected/1fa1250f-3079-4f3e-a981-515e8ec72ae8-kube-api-access-nqpxw\") pod \"image-registry-6565964545-dhkr2\" (UID: \"1fa1250f-3079-4f3e-a981-515e8ec72ae8\") " pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.806540 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.806525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" Jul 16 17:35:51.819244 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.819227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-tj4nn" Jul 16 17:35:51.850367 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.850325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:35:51.885636 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.885082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:51.976311 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.975581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t"] Jul 16 17:35:51.979575 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:51.979543 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2d0e39_8681_4c1c_9cf2_739835451b6b.slice/crio-d5b3b4e2cff042847f9e9e4657d8770dc75ba9caee4cea1b61946313d15cb826 WatchSource:0}: Error finding container d5b3b4e2cff042847f9e9e4657d8770dc75ba9caee4cea1b61946313d15cb826: Status 404 returned error can't find the container with id d5b3b4e2cff042847f9e9e4657d8770dc75ba9caee4cea1b61946313d15cb826 Jul 16 17:35:51.991681 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.991643 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-tj4nn"] Jul 16 17:35:51.992852 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.992570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfv6q" event={"ID":"98e95ef5-11f0-459f-8722-0d796d9b78d3","Type":"ContainerStarted","Data":"e829ba464366629d62fb8be59afef32914709a950f30b3ad05bbf687acf2ebfb"} Jul 16 17:35:51.993808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:51.993780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" event={"ID":"1f2d0e39-8681-4c1c-9cf2-739835451b6b","Type":"ContainerStarted","Data":"d5b3b4e2cff042847f9e9e4657d8770dc75ba9caee4cea1b61946313d15cb826"} Jul 16 17:35:51.995350 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:51.995325 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b6d468_7be6_43da_ba89_88cde195a8f3.slice/crio-665a2230632bd79356e893fdd2f32d0959e0a2e86bbe211f3cee1b73d70468e0 WatchSource:0}: Error finding container 665a2230632bd79356e893fdd2f32d0959e0a2e86bbe211f3cee1b73d70468e0: Status 404 returned error can't find the container with id 665a2230632bd79356e893fdd2f32d0959e0a2e86bbe211f3cee1b73d70468e0 Jul 16 17:35:52.015800 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:52.015726 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4887f7bd-bktzp"] Jul 16 17:35:52.020304 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:52.020260 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4989298b_df06_474c_8ee6_f65524c2987a.slice/crio-beeb36678e0f706833ef17e62273b2279256d5fb704aa7ee8e0b7cdee8388064 WatchSource:0}: Error finding container beeb36678e0f706833ef17e62273b2279256d5fb704aa7ee8e0b7cdee8388064: Status 404 returned error can't find the container with id beeb36678e0f706833ef17e62273b2279256d5fb704aa7ee8e0b7cdee8388064 Jul 16 17:35:52.041661 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:52.041547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6565964545-dhkr2"] Jul 16 17:35:52.044929 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:52.044905 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa1250f_3079_4f3e_a981_515e8ec72ae8.slice/crio-abd0fddd211b2fa73bcc5c6d422a112f1683e86859bc3bab61962b60d52f51d5 WatchSource:0}: Error finding container abd0fddd211b2fa73bcc5c6d422a112f1683e86859bc3bab61962b60d52f51d5: Status 404 returned error can't find the container with id abd0fddd211b2fa73bcc5c6d422a112f1683e86859bc3bab61962b60d52f51d5 Jul 16 17:35:52.274890 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:52.274798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:35:52.277546 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:52.277516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc-metrics-certs\") pod \"network-metrics-daemon-87b84\" (UID: \"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc\") " pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:35:52.486699 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:52.486666 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9hcv\"" Jul 16 17:35:52.494693 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:52.494667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b84" Jul 16 17:35:52.654153 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:52.654120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87b84"] Jul 16 17:35:52.657290 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:35:52.657258 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e61ff64_2ff1_4c96_ae1a_ad22e9fdddcc.slice/crio-5c05aefd529959b7fcff5fcb63f2e2d9d52ddcb20bafff8491e7293c68e35dcd WatchSource:0}: Error finding container 5c05aefd529959b7fcff5fcb63f2e2d9d52ddcb20bafff8491e7293c68e35dcd: Status 404 returned error can't find the container with id 5c05aefd529959b7fcff5fcb63f2e2d9d52ddcb20bafff8491e7293c68e35dcd Jul 16 17:35:53.000942 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:53.000856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6565964545-dhkr2" event={"ID":"1fa1250f-3079-4f3e-a981-515e8ec72ae8","Type":"ContainerStarted","Data":"8ba979f4193945ccd3188373392f194bc4e62975e0ec68247b25422d782a0e4e"} Jul 16 17:35:53.000942 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:53.000915 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6565964545-dhkr2" event={"ID":"1fa1250f-3079-4f3e-a981-515e8ec72ae8","Type":"ContainerStarted","Data":"abd0fddd211b2fa73bcc5c6d422a112f1683e86859bc3bab61962b60d52f51d5"} Jul 16 17:35:53.000942 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:53.000939 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:35:53.002552 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:53.002525 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87b84" event={"ID":"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc","Type":"ContainerStarted","Data":"5c05aefd529959b7fcff5fcb63f2e2d9d52ddcb20bafff8491e7293c68e35dcd"} Jul 16 17:35:53.003879 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:53.003853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-tj4nn" event={"ID":"04b6d468-7be6-43da-ba89-88cde195a8f3","Type":"ContainerStarted","Data":"665a2230632bd79356e893fdd2f32d0959e0a2e86bbe211f3cee1b73d70468e0"} Jul 16 17:35:53.005354 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:53.005330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4887f7bd-bktzp" event={"ID":"4989298b-df06-474c-8ee6-f65524c2987a","Type":"ContainerStarted","Data":"beeb36678e0f706833ef17e62273b2279256d5fb704aa7ee8e0b7cdee8388064"} Jul 16 17:35:53.021386 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:53.020998 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6565964545-dhkr2" podStartSLOduration=2.020979937 podStartE2EDuration="2.020979937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:53.018992985 +0000 UTC m=+131.122096836" watchObservedRunningTime="2026-07-16 17:35:53.020979937 +0000 UTC m=+131.124083776" Jul 16 17:35:59.034361 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.034318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4887f7bd-bktzp" event={"ID":"4989298b-df06-474c-8ee6-f65524c2987a","Type":"ContainerStarted","Data":"91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf"} Jul 16 17:35:59.037186 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.037113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfv6q" event={"ID":"98e95ef5-11f0-459f-8722-0d796d9b78d3","Type":"ContainerStarted","Data":"1f66a5c8c381a252b920dea17f83052441687a4d4b9b59612a9fcffe74f00c4c"} Jul 16 17:35:59.039186 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.039005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" event={"ID":"1f2d0e39-8681-4c1c-9cf2-739835451b6b","Type":"ContainerStarted","Data":"0c1dd1d541321ffc2490220f21511b022fc7ab8ac85fe2e33979810fdcbb15c9"} Jul 16 17:35:59.039347 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.039318 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" Jul 16 17:35:59.042380 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.041914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87b84" event={"ID":"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc","Type":"ContainerStarted","Data":"e6e965ec67cf922eae2f64a830aa062b1e01201199d0d306f0436a9b0ec7a174"} Jul 16 17:35:59.042380 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.041958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87b84" event={"ID":"3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc","Type":"ContainerStarted","Data":"f68efbe756f33401a8e62f9630355dc6311c95121b6b72cc1d45203f3f55d6b6"} Jul 16 17:35:59.044419 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.044398 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" Jul 16 17:35:59.052737 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.052695 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d4887f7bd-bktzp" podStartSLOduration=1.612787636 podStartE2EDuration="8.052678816s" totalImagesPullingTime="6.43989118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:59.051691993 +0000 UTC m=+137.154795845" watchObservedRunningTime="2026-07-16 17:35:59.052678816 +0000 UTC m=+137.155782645" Jul 16 17:35:59.067739 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.067615 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-5c85t" podStartSLOduration=1.58835054 podStartE2EDuration="8.067597791s" totalImagesPullingTime="6.479247251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:59.066370647 +0000 UTC m=+137.169474486" watchObservedRunningTime="2026-07-16 17:35:59.067597791 +0000 UTC m=+137.170701629" Jul 16 17:35:59.083187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.082761 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-87b84" podStartSLOduration=130.659224452 podStartE2EDuration="2m16.082741653s" totalImagesPullingTime="5.423517201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:59.081250863 +0000 UTC m=+137.184354702" watchObservedRunningTime="2026-07-16 17:35:59.082741653 +0000 UTC m=+137.185845494" Jul 16 17:35:59.699637 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.699212 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vp4tk"] Jul 16 17:35:59.704810 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.704698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.707474 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.707286 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 17:35:59.707474 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.707402 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 17:35:59.708514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.708251 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 17:35:59.708514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.708258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 17:35:59.708514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.708266 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bz9k9\"" Jul 16 17:35:59.708514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.708351 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 17:35:59.713075 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.713016 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vp4tk"] Jul 16 17:35:59.743311 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.743264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/289decec-8dcd-47b6-8dbe-e7f27e8eb881-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.743311 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.743318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.743545 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.743409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.743545 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.743526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftps6\" (UniqueName: \"kubernetes.io/projected/289decec-8dcd-47b6-8dbe-e7f27e8eb881-kube-api-access-ftps6\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.844927 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.844877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftps6\" (UniqueName: \"kubernetes.io/projected/289decec-8dcd-47b6-8dbe-e7f27e8eb881-kube-api-access-ftps6\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.845107 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.844945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/289decec-8dcd-47b6-8dbe-e7f27e8eb881-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.845107 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.844977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.845107 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.845025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.845302 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:59.845156 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 17:35:59.845302 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:35:59.845222 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-tls podName:289decec-8dcd-47b6-8dbe-e7f27e8eb881 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:00.345203742 +0000 UTC m=+138.448307560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-tls") pod "prometheus-operator-64785b6675-vp4tk" (UID: "289decec-8dcd-47b6-8dbe-e7f27e8eb881") : secret "prometheus-operator-tls" not found Jul 16 17:35:59.845811 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.845757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/289decec-8dcd-47b6-8dbe-e7f27e8eb881-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.848079 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.848054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:35:59.853924 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:35:59.853879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftps6\" (UniqueName: \"kubernetes.io/projected/289decec-8dcd-47b6-8dbe-e7f27e8eb881-kube-api-access-ftps6\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:36:00.350152 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:00.350110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:36:00.353034 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:00.352999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/289decec-8dcd-47b6-8dbe-e7f27e8eb881-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp4tk\" (UID: \"289decec-8dcd-47b6-8dbe-e7f27e8eb881\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:36:00.617808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:00.617723 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" Jul 16 17:36:01.851414 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:01.851380 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:36:01.851898 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:01.851433 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:36:01.853277 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:01.853092 2564 patch_prober.go:28] interesting pod/console-5d4887f7bd-bktzp container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body= Jul 16 17:36:01.853277 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:01.853139 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d4887f7bd-bktzp" podUID="4989298b-df06-474c-8ee6-f65524c2987a" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" Jul 16 17:36:02.410199 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:02.410175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vp4tk"] Jul 16 17:36:02.412853 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:36:02.412805 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289decec_8dcd_47b6_8dbe_e7f27e8eb881.slice/crio-28c1d427ef1697dbdc96e92e96fb800854e58fba2d8ff8e90ccb2bd5cd2a017a WatchSource:0}: Error finding container 28c1d427ef1697dbdc96e92e96fb800854e58fba2d8ff8e90ccb2bd5cd2a017a: Status 404 returned error can't find the container with id 28c1d427ef1697dbdc96e92e96fb800854e58fba2d8ff8e90ccb2bd5cd2a017a Jul 16 17:36:03.060597 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:03.060552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" event={"ID":"289decec-8dcd-47b6-8dbe-e7f27e8eb881","Type":"ContainerStarted","Data":"28c1d427ef1697dbdc96e92e96fb800854e58fba2d8ff8e90ccb2bd5cd2a017a"} Jul 16 17:36:03.062643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:03.062618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfv6q" event={"ID":"98e95ef5-11f0-459f-8722-0d796d9b78d3","Type":"ContainerStarted","Data":"057b653d1221a39bf5bd3c839157c517c45cb2c92ac9a2658b6545e76ac91f1c"} Jul 16 17:36:11.393959 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:11.393923 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:36:11.855021 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:11.854993 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:36:11.858595 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:11.858571 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:36:11.890388 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:11.890357 2564 patch_prober.go:28] interesting pod/image-registry-6565964545-dhkr2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 16 17:36:11.890531 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:11.890406 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6565964545-dhkr2" podUID="1fa1250f-3079-4f3e-a981-515e8ec72ae8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:36:13.101632 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:13.101601 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gfv6q" event={"ID":"98e95ef5-11f0-459f-8722-0d796d9b78d3","Type":"ContainerStarted","Data":"35a4781574b0488d66a669dbf207b449bbf4617634dd320686ed70662d531b86"} Jul 16 17:36:13.103066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:13.103043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-tj4nn" event={"ID":"04b6d468-7be6-43da-ba89-88cde195a8f3","Type":"ContainerStarted","Data":"f8f647ca96b315fa97c67801daf5b3f29e965c795080ecc5f0558e54fd97e802"} Jul 16 17:36:13.103159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:13.103097 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-tj4nn" Jul 16 17:36:13.104550 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:13.104531 2564 patch_prober.go:28] interesting pod/downloads-68c9cbb497-tj4nn container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body= Jul 16 17:36:13.104628 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:13.104570 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-tj4nn" podUID="04b6d468-7be6-43da-ba89-88cde195a8f3" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" Jul 16 17:36:13.120850 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:13.120786 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gfv6q" podStartSLOduration=33.707052877 podStartE2EDuration="55.120764371s" totalImagesPullingTime="21.413711494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:35:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:13.120186306 +0000 UTC m=+151.223290158" watchObservedRunningTime="2026-07-16 17:36:13.120764371 +0000 UTC m=+151.223868213" Jul 16 17:36:13.137034 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:13.136978 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-tj4nn" podStartSLOduration=1.294121042 podStartE2EDuration="22.136959515s" totalImagesPullingTime="20.842838473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:13.135985276 +0000 UTC m=+151.239089116" watchObservedRunningTime="2026-07-16 17:36:13.136959515 +0000 UTC m=+151.240063353" Jul 16 17:36:14.015338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:14.015305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6565964545-dhkr2" Jul 16 17:36:14.109950 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:14.109885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" event={"ID":"289decec-8dcd-47b6-8dbe-e7f27e8eb881","Type":"ContainerStarted","Data":"eac875026e344327d81b1d98de9bc7fd2c31a09700435870646a2cb9f85e4afc"} Jul 16 17:36:14.111385 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:14.111356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" event={"ID":"289decec-8dcd-47b6-8dbe-e7f27e8eb881","Type":"ContainerStarted","Data":"df63d0d5196953fb2f59ef8b49d43b7ad669ad72419bb895b489901118109995"} Jul 16 17:36:14.119627 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:14.119595 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-tj4nn" Jul 16 17:36:14.127470 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:14.127414 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-vp4tk" podStartSLOduration=4.468204872 podStartE2EDuration="15.127397972s" totalImagesPullingTime="10.6591931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:14.126496099 +0000 UTC m=+152.229599941" watchObservedRunningTime="2026-07-16 17:36:14.127397972 +0000 UTC m=+152.230501810" Jul 16 17:36:14.691986 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:14.691951 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d4887f7bd-bktzp"] Jul 16 17:36:16.175261 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.175209 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tkx7z"] Jul 16 17:36:16.291054 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.291023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.293677 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.293650 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 17:36:16.293813 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.293700 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 17:36:16.293813 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.293693 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 17:36:16.294048 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.294030 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zrzhn\"" Jul 16 17:36:16.389468 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d4fd-b9c1-404f-97a9-86bd2753f783-metrics-client-ca\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389634 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-wtmp\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389634 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389634 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-root\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389774 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf22\" (UniqueName: \"kubernetes.io/projected/03b6d4fd-b9c1-404f-97a9-86bd2753f783-kube-api-access-vlf22\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389774 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-sys\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389774 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-textfile\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389932 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.389932 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.389857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-tls\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.405228 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.405145 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" podUID="98415fd2-f4bd-4028-9841-f50b879fef4f" containerName="registry" containerID="cri-o://83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317" gracePeriod=30 Jul 16 17:36:16.490876 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.490837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-wtmp\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491043 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.490894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491043 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.490951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-root\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491043 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.490997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf22\" (UniqueName: \"kubernetes.io/projected/03b6d4fd-b9c1-404f-97a9-86bd2753f783-kube-api-access-vlf22\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491043 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-wtmp\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491043 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-sys\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-sys\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d4fd-b9c1-404f-97a9-86bd2753f783-root\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-textfile\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-tls\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d4fd-b9c1-404f-97a9-86bd2753f783-metrics-client-ca\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491578 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-textfile\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491688 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.491808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.491778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d4fd-b9c1-404f-97a9-86bd2753f783-metrics-client-ca\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.493906 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.493879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.493998 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.493976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d4fd-b9c1-404f-97a9-86bd2753f783-node-exporter-tls\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.500545 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.500519 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf22\" (UniqueName: \"kubernetes.io/projected/03b6d4fd-b9c1-404f-97a9-86bd2753f783-kube-api-access-vlf22\") pod \"node-exporter-tkx7z\" (UID: \"03b6d4fd-b9c1-404f-97a9-86bd2753f783\") " pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.602071 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.602042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tkx7z" Jul 16 17:36:16.637757 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:36:16.637711 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b6d4fd_b9c1_404f_97a9_86bd2753f783.slice/crio-c5c3ca5ed6a5d969f32ccb2c75d0f54da629f876b62c52ceb6673a03d024710c WatchSource:0}: Error finding container c5c3ca5ed6a5d969f32ccb2c75d0f54da629f876b62c52ceb6673a03d024710c: Status 404 returned error can't find the container with id c5c3ca5ed6a5d969f32ccb2c75d0f54da629f876b62c52ceb6673a03d024710c Jul 16 17:36:16.698317 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.698244 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:36:16.793690 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793649 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-trusted-ca\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.793877 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793726 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-installation-pull-secrets\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.793877 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793750 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-bound-sa-token\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.793877 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793776 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c79l\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-kube-api-access-2c79l\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.793877 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793845 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-image-registry-private-configuration\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.794106 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793885 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98415fd2-f4bd-4028-9841-f50b879fef4f-ca-trust-extracted\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.794106 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793924 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.794106 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.793958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-certificates\") pod \"98415fd2-f4bd-4028-9841-f50b879fef4f\" (UID: \"98415fd2-f4bd-4028-9841-f50b879fef4f\") " Jul 16 17:36:16.794106 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.794096 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:16.794298 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.794230 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-trusted-ca\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:16.794727 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.794695 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:16.796761 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.796730 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:16.797265 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.797228 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:16.797265 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.797246 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-kube-api-access-2c79l" (OuterVolumeSpecName: "kube-api-access-2c79l") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "kube-api-access-2c79l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:16.797749 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.797724 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:16.798123 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.798095 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:16.804068 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.804040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98415fd2-f4bd-4028-9841-f50b879fef4f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "98415fd2-f4bd-4028-9841-f50b879fef4f" (UID: "98415fd2-f4bd-4028-9841-f50b879fef4f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:36:16.895224 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.895153 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-installation-pull-secrets\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:16.895224 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.895192 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-bound-sa-token\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:16.895224 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.895204 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c79l\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-kube-api-access-2c79l\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:16.895224 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.895217 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98415fd2-f4bd-4028-9841-f50b879fef4f-image-registry-private-configuration\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:16.895224 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.895227 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98415fd2-f4bd-4028-9841-f50b879fef4f-ca-trust-extracted\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:16.895224 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.895236 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:16.895589 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:16.895249 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98415fd2-f4bd-4028-9841-f50b879fef4f-registry-certificates\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:17.124042 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.123572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkx7z" event={"ID":"03b6d4fd-b9c1-404f-97a9-86bd2753f783","Type":"ContainerStarted","Data":"c5c3ca5ed6a5d969f32ccb2c75d0f54da629f876b62c52ceb6673a03d024710c"} Jul 16 17:36:17.129870 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.129744 2564 generic.go:356] "Generic (PLEG): container finished" podID="98415fd2-f4bd-4028-9841-f50b879fef4f" containerID="83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317" exitCode=0 Jul 16 17:36:17.129870 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.129792 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" event={"ID":"98415fd2-f4bd-4028-9841-f50b879fef4f","Type":"ContainerDied","Data":"83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317"} Jul 16 17:36:17.129870 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.129853 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" Jul 16 17:36:17.129870 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.129868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77cb777ccb-d74d9" event={"ID":"98415fd2-f4bd-4028-9841-f50b879fef4f","Type":"ContainerDied","Data":"4f393cdc9f2796141eb5d1188cb2c0ae869a0b47fdffe4ac9640cb364244d4b9"} Jul 16 17:36:17.130167 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.129897 2564 scope.go:117] "RemoveContainer" containerID="83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317" Jul 16 17:36:17.146722 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.146696 2564 scope.go:117] "RemoveContainer" containerID="83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317" Jul 16 17:36:17.147131 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:36:17.147103 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317\": container with ID starting with 83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317 not found: ID does not exist" containerID="83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317" Jul 16 17:36:17.147236 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.147136 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317"} err="failed to get container status \"83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317\": rpc error: code = NotFound desc = could not find container \"83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317\": container with ID starting with 83b08d297615e2fe160c461e0c32120bf33ce2fa9da6e5cacbda5152d9efb317 not found: ID does not exist" Jul 16 17:36:17.161028 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.161004 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:17.161597 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.161578 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98415fd2-f4bd-4028-9841-f50b879fef4f" containerName="registry" Jul 16 17:36:17.161730 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.161717 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98415fd2-f4bd-4028-9841-f50b879fef4f" containerName="registry" Jul 16 17:36:17.161940 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.161928 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98415fd2-f4bd-4028-9841-f50b879fef4f" containerName="registry" Jul 16 17:36:17.190913 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.190880 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77cb777ccb-d74d9"] Jul 16 17:36:17.190913 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.190921 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77cb777ccb-d74d9"] Jul 16 17:36:17.191391 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.190953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:17.191391 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.191225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.193613 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.193579 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 17:36:17.194233 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194201 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 17:36:17.194359 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194295 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 17:36:17.194419 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 17:36:17.194469 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194461 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 17:36:17.194576 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194558 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 17:36:17.194649 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 17:36:17.194817 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 17:36:17.194944 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194907 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 17:36:17.195013 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.194807 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m8kn2\"" Jul 16 17:36:17.303749 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.303684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.303749 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.303735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.303749 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.303763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.303855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.303916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.303956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.304251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.304307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njnr\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-kube-api-access-6njnr\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.304341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-config-out\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.304387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-config-volume\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304723 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.304411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-web-config\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304723 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.304436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.304723 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.304467 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405667 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405667 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njnr\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-kube-api-access-6njnr\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-config-out\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-config-volume\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-web-config\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.405917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.406338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.405942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.406338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.406017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.406338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.406063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.406482 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.406422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.407261 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.406933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.407990 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.407745 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.409334 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.409182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.409334 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.409201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-config-volume\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.409334 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.409272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.409334 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.409282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.409583 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.409348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.409583 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.409462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-config-out\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.410095 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.410055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.410528 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.410510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-web-config\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.410931 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.410912 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.417177 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.417158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njnr\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-kube-api-access-6njnr\") pod \"alertmanager-main-0\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.504928 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.504884 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:36:17.875164 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:17.875050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:36:17.878322 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:36:17.878292 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2677f333_d5e4_4c13_88ca_b7efdc699684.slice/crio-dfda468f8ab41870404a117e0bb93b663933648334d5c419e93b89afbce1e9b5 WatchSource:0}: Error finding container dfda468f8ab41870404a117e0bb93b663933648334d5c419e93b89afbce1e9b5: Status 404 returned error can't find the container with id dfda468f8ab41870404a117e0bb93b663933648334d5c419e93b89afbce1e9b5 Jul 16 17:36:18.137120 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:18.137084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerStarted","Data":"dfda468f8ab41870404a117e0bb93b663933648334d5c419e93b89afbce1e9b5"} Jul 16 17:36:18.575030 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:18.574995 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98415fd2-f4bd-4028-9841-f50b879fef4f" path="/var/lib/kubelet/pods/98415fd2-f4bd-4028-9841-f50b879fef4f/volumes" Jul 16 17:36:19.059846 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.059789 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8595f96b5c-n5lhx"] Jul 16 17:36:19.096154 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.096108 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8595f96b5c-n5lhx"] Jul 16 17:36:19.096315 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.096235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.099127 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.098847 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 17:36:19.099127 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.098880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2dq9d\"" Jul 16 17:36:19.099127 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.098891 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 17:36:19.099127 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.098982 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3k6dms3dar49s\"" Jul 16 17:36:19.099422 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.099232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 17:36:19.099422 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.099329 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 17:36:19.108369 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.108348 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 17:36:19.143178 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.143145 2564 generic.go:356] "Generic (PLEG): container finished" podID="03b6d4fd-b9c1-404f-97a9-86bd2753f783" containerID="162773106cd2c7e3fe17e9655cd8ee0c6a267b38dbee8929c2a64a766a9021f0" exitCode=0 Jul 16 17:36:19.143329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.143235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkx7z" event={"ID":"03b6d4fd-b9c1-404f-97a9-86bd2753f783","Type":"ContainerDied","Data":"162773106cd2c7e3fe17e9655cd8ee0c6a267b38dbee8929c2a64a766a9021f0"} Jul 16 17:36:19.225137 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htdw\" (UniqueName: \"kubernetes.io/projected/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-kube-api-access-9htdw\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.225281 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-grpc-tls\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.225281 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.225382 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.225493 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.225719 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-metrics-client-ca\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.225812 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.225898 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.225821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-tls\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.327199 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.327169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.327332 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.327248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-metrics-client-ca\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.327332 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.327287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.327332 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.327320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-tls\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.327489 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.327369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htdw\" (UniqueName: \"kubernetes.io/projected/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-kube-api-access-9htdw\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.327489 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.327397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-grpc-tls\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.327489 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.327423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.328110 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.328084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-metrics-client-ca\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.328235 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.328148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.330193 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.330125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-tls\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.330307 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.330268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.330577 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.330523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-grpc-tls\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.330960 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.330919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.331209 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.331185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.331690 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.331669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.337150 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.337123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htdw\" (UniqueName: \"kubernetes.io/projected/dfd5e33b-3b9f-40ea-9aef-50c4867e68e2-kube-api-access-9htdw\") pod \"thanos-querier-8595f96b5c-n5lhx\" (UID: \"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2\") " pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.408133 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.408099 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:19.698460 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:19.698434 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8595f96b5c-n5lhx"] Jul 16 17:36:19.702323 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:36:19.702294 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd5e33b_3b9f_40ea_9aef_50c4867e68e2.slice/crio-19ff9d7f239583f2e13920e06f7617ead69e4c37486cb620705c04c6d7e43657 WatchSource:0}: Error finding container 19ff9d7f239583f2e13920e06f7617ead69e4c37486cb620705c04c6d7e43657: Status 404 returned error can't find the container with id 19ff9d7f239583f2e13920e06f7617ead69e4c37486cb620705c04c6d7e43657 Jul 16 17:36:20.150018 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.149975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkx7z" event={"ID":"03b6d4fd-b9c1-404f-97a9-86bd2753f783","Type":"ContainerStarted","Data":"6944ed938a713886a8aed9e557f8e0357a924fbd0c56d89e10a19575ac6f77c6"} Jul 16 17:36:20.150211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.150057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkx7z" event={"ID":"03b6d4fd-b9c1-404f-97a9-86bd2753f783","Type":"ContainerStarted","Data":"80124516fd959fabbe3b15d61a36471136655d29fcdcbd1f457e4f22d0ffab2c"} Jul 16 17:36:20.151429 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.151392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" event={"ID":"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2","Type":"ContainerStarted","Data":"19ff9d7f239583f2e13920e06f7617ead69e4c37486cb620705c04c6d7e43657"} Jul 16 17:36:20.153070 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.153043 2564 generic.go:356] "Generic (PLEG): container finished" podID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerID="ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312" exitCode=0 Jul 16 17:36:20.153162 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.153096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312"} Jul 16 17:36:20.172628 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.172576 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tkx7z" podStartSLOduration=2.555101599 podStartE2EDuration="4.172561423s" totalImagesPullingTime="1.534343271s" totalInitContainerRuntime="83.116553ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:20.170924482 +0000 UTC m=+158.274028321" watchObservedRunningTime="2026-07-16 17:36:20.172561423 +0000 UTC m=+158.275665261" Jul 16 17:36:20.503055 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.502973 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8f844764d-fwzz5"] Jul 16 17:36:20.517939 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.517906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.518792 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.518763 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8f844764d-fwzz5"] Jul 16 17:36:20.520352 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.520322 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 17:36:20.520462 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.520322 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 17:36:20.521285 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.521263 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 17:36:20.521386 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.521289 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3si3o6luiejch\"" Jul 16 17:36:20.521480 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.521460 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zqmsh\"" Jul 16 17:36:20.521723 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.521705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 17:36:20.640666 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.640624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jvcz\" (UniqueName: \"kubernetes.io/projected/e2c9404d-97d8-4779-bf17-7d19fffad34d-kube-api-access-7jvcz\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.640885 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.640685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c9404d-97d8-4779-bf17-7d19fffad34d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.640885 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.640795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-secret-metrics-server-tls\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.640885 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.640842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e2c9404d-97d8-4779-bf17-7d19fffad34d-metrics-server-audit-profiles\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.641013 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.640898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-client-ca-bundle\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.641013 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.640986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-secret-metrics-server-client-certs\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.641099 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.641015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e2c9404d-97d8-4779-bf17-7d19fffad34d-audit-log\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.741780 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.741736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-secret-metrics-server-client-certs\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.741780 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.741784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e2c9404d-97d8-4779-bf17-7d19fffad34d-audit-log\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.742622 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.741818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jvcz\" (UniqueName: \"kubernetes.io/projected/e2c9404d-97d8-4779-bf17-7d19fffad34d-kube-api-access-7jvcz\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.742622 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.742096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c9404d-97d8-4779-bf17-7d19fffad34d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.743820 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.742979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-secret-metrics-server-tls\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.743820 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.743030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e2c9404d-97d8-4779-bf17-7d19fffad34d-metrics-server-audit-profiles\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.743820 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.743141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c9404d-97d8-4779-bf17-7d19fffad34d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.743820 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.743705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-client-ca-bundle\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.743820 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.743775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e2c9404d-97d8-4779-bf17-7d19fffad34d-audit-log\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.744465 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.744419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e2c9404d-97d8-4779-bf17-7d19fffad34d-metrics-server-audit-profiles\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.746626 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.746584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-secret-metrics-server-tls\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.746725 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.746680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-secret-metrics-server-client-certs\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.747627 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.747589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2c9404d-97d8-4779-bf17-7d19fffad34d-client-ca-bundle\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.758382 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.758178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jvcz\" (UniqueName: \"kubernetes.io/projected/e2c9404d-97d8-4779-bf17-7d19fffad34d-kube-api-access-7jvcz\") pod \"metrics-server-8f844764d-fwzz5\" (UID: \"e2c9404d-97d8-4779-bf17-7d19fffad34d\") " pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:20.830191 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:20.829918 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:21.047023 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.046987 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8f844764d-fwzz5"] Jul 16 17:36:21.052496 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:36:21.052447 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c9404d_97d8_4779_bf17_7d19fffad34d.slice/crio-697eb5b60f41745d97795f83798a8024be157685e85c5f2fe73f51042a40e9cc WatchSource:0}: Error finding container 697eb5b60f41745d97795f83798a8024be157685e85c5f2fe73f51042a40e9cc: Status 404 returned error can't find the container with id 697eb5b60f41745d97795f83798a8024be157685e85c5f2fe73f51042a40e9cc Jul 16 17:36:21.161858 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.161752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" event={"ID":"e2c9404d-97d8-4779-bf17-7d19fffad34d","Type":"ContainerStarted","Data":"697eb5b60f41745d97795f83798a8024be157685e85c5f2fe73f51042a40e9cc"} Jul 16 17:36:21.310031 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:36:21.309949 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z54xl" podUID="41658552-fcc5-420f-80c1-9078dfb115a6" Jul 16 17:36:21.828616 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.828576 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79784fc5fb-xfhml"] Jul 16 17:36:21.852619 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.852588 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79784fc5fb-xfhml"] Jul 16 17:36:21.852801 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.852692 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:21.860178 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.860150 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 17:36:21.956979 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.956943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-oauth-config\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:21.956979 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.956986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-service-ca\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:21.957197 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.957055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-console-config\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:21.957197 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.957084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-serving-cert\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:21.957197 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.957099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-trusted-ca-bundle\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:21.957197 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.957167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf85b\" (UniqueName: \"kubernetes.io/projected/463a87cc-aeb4-426e-9842-d3b756000fef-kube-api-access-qf85b\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:21.957315 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:21.957201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-oauth-serving-cert\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.058288 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.058249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf85b\" (UniqueName: \"kubernetes.io/projected/463a87cc-aeb4-426e-9842-d3b756000fef-kube-api-access-qf85b\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.058288 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.058286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-oauth-serving-cert\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.058520 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.058332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-oauth-config\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.058520 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.058349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-service-ca\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.058520 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.058395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-console-config\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.058653 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.058565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-serving-cert\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.058653 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.058597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-trusted-ca-bundle\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.059142 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.059117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-oauth-serving-cert\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.059275 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.059186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-console-config\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.059275 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.059225 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-service-ca\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.059594 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.059575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-trusted-ca-bundle\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.060923 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.060902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-oauth-config\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.061011 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.060985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-serving-cert\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.069667 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.069624 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf85b\" (UniqueName: \"kubernetes.io/projected/463a87cc-aeb4-426e-9842-d3b756000fef-kube-api-access-qf85b\") pod \"console-79784fc5fb-xfhml\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.165450 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.165417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:22.166705 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.166683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z54xl" Jul 16 17:36:22.329187 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:22.329148 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79784fc5fb-xfhml"] Jul 16 17:36:22.331424 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:36:22.331395 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463a87cc_aeb4_426e_9842_d3b756000fef.slice/crio-4d3721fe147010ca20fe6cd371d58f37d50c6e8ba8956d1ec052d758257387c4 WatchSource:0}: Error finding container 4d3721fe147010ca20fe6cd371d58f37d50c6e8ba8956d1ec052d758257387c4: Status 404 returned error can't find the container with id 4d3721fe147010ca20fe6cd371d58f37d50c6e8ba8956d1ec052d758257387c4 Jul 16 17:36:23.175933 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.175856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerStarted","Data":"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0"} Jul 16 17:36:23.176502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.175991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerStarted","Data":"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd"} Jul 16 17:36:23.176502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.176033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerStarted","Data":"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068"} Jul 16 17:36:23.176502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.176073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerStarted","Data":"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d"} Jul 16 17:36:23.176502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.176112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerStarted","Data":"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300"} Jul 16 17:36:23.178224 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.178196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79784fc5fb-xfhml" event={"ID":"463a87cc-aeb4-426e-9842-d3b756000fef","Type":"ContainerStarted","Data":"00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078"} Jul 16 17:36:23.178365 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.178244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79784fc5fb-xfhml" event={"ID":"463a87cc-aeb4-426e-9842-d3b756000fef","Type":"ContainerStarted","Data":"4d3721fe147010ca20fe6cd371d58f37d50c6e8ba8956d1ec052d758257387c4"} Jul 16 17:36:23.198896 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:23.198821 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79784fc5fb-xfhml" podStartSLOduration=2.198798579 podStartE2EDuration="2.198798579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:23.197719638 +0000 UTC m=+161.300823479" watchObservedRunningTime="2026-07-16 17:36:23.198798579 +0000 UTC m=+161.301902418" Jul 16 17:36:24.184403 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:24.184369 2564 generic.go:356] "Generic (PLEG): container finished" podID="80e75ef9-a0a6-4037-8db0-3cd0c08369c6" containerID="8075c4a99eaf6916756847cd7ddfde32949b44edca02d17c0bdfa1401609e596" exitCode=0 Jul 16 17:36:24.184815 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:24.184436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" event={"ID":"80e75ef9-a0a6-4037-8db0-3cd0c08369c6","Type":"ContainerDied","Data":"8075c4a99eaf6916756847cd7ddfde32949b44edca02d17c0bdfa1401609e596"} Jul 16 17:36:24.184909 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:24.184894 2564 scope.go:117] "RemoveContainer" containerID="8075c4a99eaf6916756847cd7ddfde32949b44edca02d17c0bdfa1401609e596" Jul 16 17:36:25.195846 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:25.195786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-dhckd" event={"ID":"80e75ef9-a0a6-4037-8db0-3cd0c08369c6","Type":"ContainerStarted","Data":"6d596bc8eb16e2d1956245150f3c128725406604129e234fb6441817b4a04a87"} Jul 16 17:36:26.201336 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.201294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" event={"ID":"e2c9404d-97d8-4779-bf17-7d19fffad34d","Type":"ContainerStarted","Data":"8b8df5ebcc85a259ee7f788b091472440656d7f06f5bd9d7b05e26ec96fceafa"} Jul 16 17:36:26.201956 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.201934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:36:26.204499 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.204472 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41658552-fcc5-420f-80c1-9078dfb115a6-metrics-tls\") pod \"dns-default-z54xl\" (UID: \"41658552-fcc5-420f-80c1-9078dfb115a6\") " pod="openshift-dns/dns-default-z54xl" Jul 16 17:36:26.204988 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.204967 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerStarted","Data":"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc"} Jul 16 17:36:26.208211 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.208186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" event={"ID":"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2","Type":"ContainerStarted","Data":"6d8fb98d2ec009073c9028748e00a80d81e9d41e2fe3ad245a76dded90ec07b5"} Jul 16 17:36:26.208291 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.208250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" event={"ID":"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2","Type":"ContainerStarted","Data":"525645eaa671fc24740f90bd1ff78cd76a40c6afc61ce1aaa6fe011a77656da7"} Jul 16 17:36:26.208329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.208291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" event={"ID":"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2","Type":"ContainerStarted","Data":"b5d367872b0134f37176845734f94fbdac248a855231dfae2cdc5e693b0fc03d"} Jul 16 17:36:26.208365 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.208334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" event={"ID":"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2","Type":"ContainerStarted","Data":"2f18895f90261a94c9e527971c6abc118389a61e6cbf7e5a167afe46674a527a"} Jul 16 17:36:26.208395 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.208366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:26.208395 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.208390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" event={"ID":"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2","Type":"ContainerStarted","Data":"e6353d9eca8cf02003890e1db605b5e8bd78eb04f3e46ddbb73b139b8abdbf39"} Jul 16 17:36:26.208455 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.208423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" event={"ID":"dfd5e33b-3b9f-40ea-9aef-50c4867e68e2","Type":"ContainerStarted","Data":"9ef3aa4d0cb067e7e962d84c2538cc843c66cd9f7f84310173c5aa77e9c85b2a"} Jul 16 17:36:26.220753 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.220696 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" podStartSLOduration=2.174830784 podStartE2EDuration="6.220682692s" totalImagesPullingTime="4.045851908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:26.219774647 +0000 UTC m=+164.322878497" watchObservedRunningTime="2026-07-16 17:36:26.220682692 +0000 UTC m=+164.323786532" Jul 16 17:36:26.251283 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.251190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.236439891 podStartE2EDuration="9.25117075s" totalImagesPullingTime="5.994751742s" totalInitContainerRuntime="19.979117ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:26.249425162 +0000 UTC m=+164.352529002" watchObservedRunningTime="2026-07-16 17:36:26.25117075 +0000 UTC m=+164.354274589" Jul 16 17:36:26.273435 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.273370 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" podStartSLOduration=1.523507399 podStartE2EDuration="7.2733512s" totalImagesPullingTime="5.749843801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:26.27244323 +0000 UTC m=+164.375547071" watchObservedRunningTime="2026-07-16 17:36:26.2733512 +0000 UTC m=+164.376455038" Jul 16 17:36:26.370040 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.370012 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ntp5m\"" Jul 16 17:36:26.378837 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.378808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z54xl" Jul 16 17:36:26.529822 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:26.529798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z54xl"] Jul 16 17:36:26.532291 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:36:26.532257 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41658552_fcc5_420f_80c1_9078dfb115a6.slice/crio-0c2d84ab18f8cb49784f164fc8577accbd5396a7be497fb6d5a7810fbb3ff586 WatchSource:0}: Error finding container 0c2d84ab18f8cb49784f164fc8577accbd5396a7be497fb6d5a7810fbb3ff586: Status 404 returned error can't find the container with id 0c2d84ab18f8cb49784f164fc8577accbd5396a7be497fb6d5a7810fbb3ff586 Jul 16 17:36:27.214580 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:27.214535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z54xl" event={"ID":"41658552-fcc5-420f-80c1-9078dfb115a6","Type":"ContainerStarted","Data":"0c2d84ab18f8cb49784f164fc8577accbd5396a7be497fb6d5a7810fbb3ff586"} Jul 16 17:36:29.223437 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:29.223404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z54xl" event={"ID":"41658552-fcc5-420f-80c1-9078dfb115a6","Type":"ContainerStarted","Data":"ae7fc6994031de02ba21e5166ee73833fa16cc99946c9996a7f73944106be81d"} Jul 16 17:36:29.223902 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:29.223454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z54xl" event={"ID":"41658552-fcc5-420f-80c1-9078dfb115a6","Type":"ContainerStarted","Data":"e1615197c46c2531cf3cae3af1509aec7ad5c07b7688bcb42fef20ad9883ca14"} Jul 16 17:36:29.223902 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:29.223481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z54xl" Jul 16 17:36:29.244883 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:29.243951 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z54xl" podStartSLOduration=129.277807218 podStartE2EDuration="2m11.243926508s" totalImagesPullingTime="1.96611929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:29.241144633 +0000 UTC m=+167.344248492" watchObservedRunningTime="2026-07-16 17:36:29.243926508 +0000 UTC m=+167.347030347" Jul 16 17:36:32.166123 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:32.166079 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:32.166544 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:32.166147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:32.171944 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:32.171920 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:32.220200 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:32.220166 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8595f96b5c-n5lhx" Jul 16 17:36:32.238526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:32.238500 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:36:39.230522 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:39.230450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z54xl" Jul 16 17:36:39.717032 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:39.716993 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d4887f7bd-bktzp" podUID="4989298b-df06-474c-8ee6-f65524c2987a" containerName="console" containerID="cri-o://91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf" gracePeriod=15 Jul 16 17:36:39.986179 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:39.986108 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:36:40.020423 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020397 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6dk9\" (UniqueName: \"kubernetes.io/projected/4989298b-df06-474c-8ee6-f65524c2987a-kube-api-access-b6dk9\") pod \"4989298b-df06-474c-8ee6-f65524c2987a\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " Jul 16 17:36:40.020577 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020432 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-console-config\") pod \"4989298b-df06-474c-8ee6-f65524c2987a\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " Jul 16 17:36:40.020577 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020451 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-oauth-serving-cert\") pod \"4989298b-df06-474c-8ee6-f65524c2987a\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " Jul 16 17:36:40.020577 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020481 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-oauth-config\") pod \"4989298b-df06-474c-8ee6-f65524c2987a\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " Jul 16 17:36:40.020577 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020519 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-serving-cert\") pod \"4989298b-df06-474c-8ee6-f65524c2987a\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " Jul 16 17:36:40.020577 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020538 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-service-ca\") pod \"4989298b-df06-474c-8ee6-f65524c2987a\" (UID: \"4989298b-df06-474c-8ee6-f65524c2987a\") " Jul 16 17:36:40.020983 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020950 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4989298b-df06-474c-8ee6-f65524c2987a" (UID: "4989298b-df06-474c-8ee6-f65524c2987a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:40.020983 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.020963 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-console-config" (OuterVolumeSpecName: "console-config") pod "4989298b-df06-474c-8ee6-f65524c2987a" (UID: "4989298b-df06-474c-8ee6-f65524c2987a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:40.021146 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.021026 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-service-ca" (OuterVolumeSpecName: "service-ca") pod "4989298b-df06-474c-8ee6-f65524c2987a" (UID: "4989298b-df06-474c-8ee6-f65524c2987a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:36:40.022754 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.022722 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4989298b-df06-474c-8ee6-f65524c2987a" (UID: "4989298b-df06-474c-8ee6-f65524c2987a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:40.022877 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.022773 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4989298b-df06-474c-8ee6-f65524c2987a" (UID: "4989298b-df06-474c-8ee6-f65524c2987a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:36:40.022877 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.022780 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4989298b-df06-474c-8ee6-f65524c2987a-kube-api-access-b6dk9" (OuterVolumeSpecName: "kube-api-access-b6dk9") pod "4989298b-df06-474c-8ee6-f65524c2987a" (UID: "4989298b-df06-474c-8ee6-f65524c2987a"). InnerVolumeSpecName "kube-api-access-b6dk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:36:40.121363 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.121331 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-oauth-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:40.121363 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.121359 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4989298b-df06-474c-8ee6-f65524c2987a-console-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:40.121363 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.121369 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-service-ca\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:40.121576 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.121378 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6dk9\" (UniqueName: \"kubernetes.io/projected/4989298b-df06-474c-8ee6-f65524c2987a-kube-api-access-b6dk9\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:40.121576 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.121387 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-console-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:40.121576 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.121395 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4989298b-df06-474c-8ee6-f65524c2987a-oauth-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:36:40.269076 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.268989 2564 generic.go:356] "Generic (PLEG): container finished" podID="4989298b-df06-474c-8ee6-f65524c2987a" containerID="91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf" exitCode=0 Jul 16 17:36:40.269076 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.269042 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4887f7bd-bktzp" Jul 16 17:36:40.269552 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.269080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4887f7bd-bktzp" event={"ID":"4989298b-df06-474c-8ee6-f65524c2987a","Type":"ContainerDied","Data":"91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf"} Jul 16 17:36:40.269552 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.269154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4887f7bd-bktzp" event={"ID":"4989298b-df06-474c-8ee6-f65524c2987a","Type":"ContainerDied","Data":"beeb36678e0f706833ef17e62273b2279256d5fb704aa7ee8e0b7cdee8388064"} Jul 16 17:36:40.269552 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.269180 2564 scope.go:117] "RemoveContainer" containerID="91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf" Jul 16 17:36:40.277926 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.277906 2564 scope.go:117] "RemoveContainer" containerID="91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf" Jul 16 17:36:40.278175 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:36:40.278159 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf\": container with ID starting with 91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf not found: ID does not exist" containerID="91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf" Jul 16 17:36:40.278213 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.278184 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf"} err="failed to get container status \"91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf\": rpc error: code = NotFound desc = could not find container \"91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf\": container with ID starting with 91e0e2df619541e24f5e268a1b3433d7ef1943d18a1be5aceab656f9adff58cf not found: ID does not exist" Jul 16 17:36:40.290111 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.290085 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d4887f7bd-bktzp"] Jul 16 17:36:40.298847 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.298810 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d4887f7bd-bktzp"] Jul 16 17:36:40.573339 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.573299 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4989298b-df06-474c-8ee6-f65524c2987a" path="/var/lib/kubelet/pods/4989298b-df06-474c-8ee6-f65524c2987a/volumes" Jul 16 17:36:40.830047 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.829971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:36:40.830047 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:36:40.830030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:37:00.835952 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:00.835912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:37:00.839879 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:00.839857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8f844764d-fwzz5" Jul 16 17:37:36.398199 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:36.398166 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:37:36.398713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:36.398441 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="alertmanager" containerID="cri-o://1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300" gracePeriod=120 Jul 16 17:37:36.398713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:36.398541 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-web" containerID="cri-o://6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068" gracePeriod=120 Jul 16 17:37:36.398713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:36.398569 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy" containerID="cri-o://007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd" gracePeriod=120 Jul 16 17:37:36.398713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:36.398572 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="prom-label-proxy" containerID="cri-o://214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc" gracePeriod=120 Jul 16 17:37:36.398713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:36.398587 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="config-reloader" containerID="cri-o://b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d" gracePeriod=120 Jul 16 17:37:36.398713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:36.398523 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-metric" containerID="cri-o://8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0" gracePeriod=120 Jul 16 17:37:37.485008 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.484981 2564 generic.go:356] "Generic (PLEG): container finished" podID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerID="214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc" exitCode=0 Jul 16 17:37:37.485008 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.485008 2564 generic.go:356] "Generic (PLEG): container finished" podID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerID="007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd" exitCode=0 Jul 16 17:37:37.485329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.485018 2564 generic.go:356] "Generic (PLEG): container finished" podID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerID="b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d" exitCode=0 Jul 16 17:37:37.485329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.485024 2564 generic.go:356] "Generic (PLEG): container finished" podID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerID="1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300" exitCode=0 Jul 16 17:37:37.485329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.485050 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc"} Jul 16 17:37:37.485329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.485113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd"} Jul 16 17:37:37.485329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.485139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d"} Jul 16 17:37:37.485329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.485165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300"} Jul 16 17:37:37.646921 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.646896 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:37.696429 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696383 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-main-db\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.696612 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-web\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.696612 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696491 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-config-volume\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.696612 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696512 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-trusted-ca-bundle\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.696612 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696533 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.696612 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-config-out\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.696612 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6njnr\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-kube-api-access-6njnr\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.697018 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696635 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-web-config\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.697018 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-main-tls\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.697018 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.696722 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.697250 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.697206 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:37.698212 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.697417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-tls-assets\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.698212 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.697463 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-cluster-tls-config\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.698212 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.697497 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-metrics-client-ca\") pod \"2677f333-d5e4-4c13-88ca-b7efdc699684\" (UID: \"2677f333-d5e4-4c13-88ca-b7efdc699684\") " Jul 16 17:37:37.698212 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.697797 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:37:37.698212 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.697822 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-main-db\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.698212 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.698158 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:37:37.699733 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.699681 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:37:37.700799 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.700765 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:37:37.700914 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.700793 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:37:37.700914 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.700819 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-kube-api-access-6njnr" (OuterVolumeSpecName: "kube-api-access-6njnr") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "kube-api-access-6njnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:37:37.701017 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.700950 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:37:37.701190 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.701166 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-config-out" (OuterVolumeSpecName: "config-out") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:37:37.701314 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.701288 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-config-volume" (OuterVolumeSpecName: "config-volume") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:37:37.701405 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.701392 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:37:37.705514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.705488 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:37:37.712409 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.712353 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-web-config" (OuterVolumeSpecName: "web-config") pod "2677f333-d5e4-4c13-88ca-b7efdc699684" (UID: "2677f333-d5e4-4c13-88ca-b7efdc699684"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:37:37.798928 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798889 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6njnr\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-kube-api-access-6njnr\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.798928 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798921 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-web-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.798928 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798931 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-main-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798942 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798951 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2677f333-d5e4-4c13-88ca-b7efdc699684-tls-assets\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798962 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-cluster-tls-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798971 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-metrics-client-ca\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798979 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798990 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-config-volume\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.798998 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2677f333-d5e4-4c13-88ca-b7efdc699684-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.799006 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2677f333-d5e4-4c13-88ca-b7efdc699684-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:37.799159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:37.799014 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2677f333-d5e4-4c13-88ca-b7efdc699684-config-out\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:37:38.492311 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.492277 2564 generic.go:356] "Generic (PLEG): container finished" podID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerID="8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0" exitCode=0 Jul 16 17:37:38.492311 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.492308 2564 generic.go:356] "Generic (PLEG): container finished" podID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerID="6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068" exitCode=0 Jul 16 17:37:38.492779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.492361 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.492779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.492365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0"} Jul 16 17:37:38.492779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.492453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068"} Jul 16 17:37:38.492779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.492499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2677f333-d5e4-4c13-88ca-b7efdc699684","Type":"ContainerDied","Data":"dfda468f8ab41870404a117e0bb93b663933648334d5c419e93b89afbce1e9b5"} Jul 16 17:37:38.492779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.492524 2564 scope.go:117] "RemoveContainer" containerID="214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc" Jul 16 17:37:38.501333 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.501313 2564 scope.go:117] "RemoveContainer" containerID="8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0" Jul 16 17:37:38.510861 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.510675 2564 scope.go:117] "RemoveContainer" containerID="007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd" Jul 16 17:37:38.517873 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.517845 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:37:38.518327 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.518313 2564 scope.go:117] "RemoveContainer" containerID="6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068" Jul 16 17:37:38.525951 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.525191 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:37:38.530185 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.530165 2564 scope.go:117] "RemoveContainer" containerID="b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d" Jul 16 17:37:38.537599 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.537575 2564 scope.go:117] "RemoveContainer" containerID="1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300" Jul 16 17:37:38.545285 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.545258 2564 scope.go:117] "RemoveContainer" containerID="ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312" Jul 16 17:37:38.550162 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550140 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:37:38.550462 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550449 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="init-config-reloader" Jul 16 17:37:38.550500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550464 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="init-config-reloader" Jul 16 17:37:38.550500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550481 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4989298b-df06-474c-8ee6-f65524c2987a" containerName="console" Jul 16 17:37:38.550500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550486 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4989298b-df06-474c-8ee6-f65524c2987a" containerName="console" Jul 16 17:37:38.550500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550494 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="config-reloader" Jul 16 17:37:38.550500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550500 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="config-reloader" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550506 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="prom-label-proxy" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550511 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="prom-label-proxy" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550519 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="alertmanager" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550523 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="alertmanager" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550530 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-web" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550535 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-web" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550540 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550545 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550553 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-metric" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550558 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-metric" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550601 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="alertmanager" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550610 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="prom-label-proxy" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550616 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-web" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550622 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4989298b-df06-474c-8ee6-f65524c2987a" containerName="console" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550628 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="config-reloader" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550634 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy-metric" Jul 16 17:37:38.550643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.550641 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" containerName="kube-rbac-proxy" Jul 16 17:37:38.553785 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.553764 2564 scope.go:117] "RemoveContainer" containerID="214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc" Jul 16 17:37:38.554271 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:37:38.554241 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc\": container with ID starting with 214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc not found: ID does not exist" containerID="214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc" Jul 16 17:37:38.554355 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.554281 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc"} err="failed to get container status \"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc\": rpc error: code = NotFound desc = could not find container \"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc\": container with ID starting with 214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc not found: ID does not exist" Jul 16 17:37:38.554355 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.554307 2564 scope.go:117] "RemoveContainer" containerID="8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0" Jul 16 17:37:38.554442 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.554372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.554696 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:37:38.554601 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0\": container with ID starting with 8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0 not found: ID does not exist" containerID="8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0" Jul 16 17:37:38.554696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.554642 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0"} err="failed to get container status \"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0\": rpc error: code = NotFound desc = could not find container \"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0\": container with ID starting with 8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0 not found: ID does not exist" Jul 16 17:37:38.554696 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.554694 2564 scope.go:117] "RemoveContainer" containerID="007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd" Jul 16 17:37:38.555012 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:37:38.554987 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd\": container with ID starting with 007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd not found: ID does not exist" containerID="007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd" Jul 16 17:37:38.555065 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555024 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd"} err="failed to get container status \"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd\": rpc error: code = NotFound desc = could not find container \"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd\": container with ID starting with 007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd not found: ID does not exist" Jul 16 17:37:38.555065 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555041 2564 scope.go:117] "RemoveContainer" containerID="6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068" Jul 16 17:37:38.555284 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:37:38.555264 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068\": container with ID starting with 6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068 not found: ID does not exist" containerID="6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068" Jul 16 17:37:38.555329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555291 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068"} err="failed to get container status \"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068\": rpc error: code = NotFound desc = could not find container \"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068\": container with ID starting with 6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068 not found: ID does not exist" Jul 16 17:37:38.555329 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555309 2564 scope.go:117] "RemoveContainer" containerID="b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d" Jul 16 17:37:38.555558 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:37:38.555539 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d\": container with ID starting with b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d not found: ID does not exist" containerID="b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d" Jul 16 17:37:38.555608 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555560 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d"} err="failed to get container status \"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d\": rpc error: code = NotFound desc = could not find container \"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d\": container with ID starting with b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d not found: ID does not exist" Jul 16 17:37:38.555608 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555575 2564 scope.go:117] "RemoveContainer" containerID="1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300" Jul 16 17:37:38.555785 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:37:38.555769 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300\": container with ID starting with 1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300 not found: ID does not exist" containerID="1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300" Jul 16 17:37:38.555851 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555789 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300"} err="failed to get container status \"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300\": rpc error: code = NotFound desc = could not find container \"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300\": container with ID starting with 1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300 not found: ID does not exist" Jul 16 17:37:38.555851 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.555802 2564 scope.go:117] "RemoveContainer" containerID="ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312" Jul 16 17:37:38.556050 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:37:38.556033 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312\": container with ID starting with ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312 not found: ID does not exist" containerID="ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312" Jul 16 17:37:38.556087 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556057 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312"} err="failed to get container status \"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312\": rpc error: code = NotFound desc = could not find container \"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312\": container with ID starting with ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312 not found: ID does not exist" Jul 16 17:37:38.556087 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556071 2564 scope.go:117] "RemoveContainer" containerID="214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc" Jul 16 17:37:38.556294 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556272 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc"} err="failed to get container status \"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc\": rpc error: code = NotFound desc = could not find container \"214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc\": container with ID starting with 214332498dfffa5762228fb1685ca885d9a703771cfa91af169fd1da079ec2bc not found: ID does not exist" Jul 16 17:37:38.556340 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556297 2564 scope.go:117] "RemoveContainer" containerID="8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0" Jul 16 17:37:38.556561 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556536 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0"} err="failed to get container status \"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0\": rpc error: code = NotFound desc = could not find container \"8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0\": container with ID starting with 8cb6a9037b64377c5d5a859e532ba60f06a7357ad6a5787bde76a5baae3a98f0 not found: ID does not exist" Jul 16 17:37:38.556561 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556560 2564 scope.go:117] "RemoveContainer" containerID="007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd" Jul 16 17:37:38.556784 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556767 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 17:37:38.556864 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556793 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd"} err="failed to get container status \"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd\": rpc error: code = NotFound desc = could not find container \"007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd\": container with ID starting with 007961b88a158a928167f6b7290bf37dc7ca65269a7c6ba29359944a36a1d7dd not found: ID does not exist" Jul 16 17:37:38.556864 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556815 2564 scope.go:117] "RemoveContainer" containerID="6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068" Jul 16 17:37:38.556981 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.556951 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 17:37:38.557108 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557074 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m8kn2\"" Jul 16 17:37:38.557173 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557123 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 17:37:38.557173 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 17:37:38.557270 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 17:37:38.557270 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557217 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 17:37:38.557270 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557165 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 17:37:38.557407 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557388 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068"} err="failed to get container status \"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068\": rpc error: code = NotFound desc = could not find container \"6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068\": container with ID starting with 6e012010ed77c640e7c3bcbedc979caca131cd0e0f0f5751120c07bf75606068 not found: ID does not exist" Jul 16 17:37:38.557463 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557410 2564 scope.go:117] "RemoveContainer" containerID="b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d" Jul 16 17:37:38.557514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557461 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 17:37:38.558038 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557779 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d"} err="failed to get container status \"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d\": rpc error: code = NotFound desc = could not find container \"b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d\": container with ID starting with b0b4603fdf9539bc4ec89830bdf9209a48d03df8f5f82b6677007b02a475931d not found: ID does not exist" Jul 16 17:37:38.558038 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.557810 2564 scope.go:117] "RemoveContainer" containerID="1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300" Jul 16 17:37:38.558368 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.558277 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300"} err="failed to get container status \"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300\": rpc error: code = NotFound desc = could not find container \"1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300\": container with ID starting with 1ad3f8796f3a97e4e3033e00572b5f6e6087077a91f7bfc915698cccfa2f4300 not found: ID does not exist" Jul 16 17:37:38.558368 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.558301 2564 scope.go:117] "RemoveContainer" containerID="ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312" Jul 16 17:37:38.558634 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.558588 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312"} err="failed to get container status \"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312\": rpc error: code = NotFound desc = could not find container \"ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312\": container with ID starting with ff89f11ca7e6a5697acc171914d0fb8ba7e0cabf75549f6b0e1171f80e0e3312 not found: ID does not exist" Jul 16 17:37:38.562859 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.562821 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 17:37:38.575038 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.575007 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2677f333-d5e4-4c13-88ca-b7efdc699684" path="/var/lib/kubelet/pods/2677f333-d5e4-4c13-88ca-b7efdc699684/volumes" Jul 16 17:37:38.575520 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.575504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:37:38.605935 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.605899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df435d78-d969-405c-b381-128a788e2fb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606131 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.605949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606131 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606131 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df435d78-d969-405c-b381-128a788e2fb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606131 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606293 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606293 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df435d78-d969-405c-b381-128a788e2fb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606293 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df435d78-d969-405c-b381-128a788e2fb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606293 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcd4\" (UniqueName: \"kubernetes.io/projected/df435d78-d969-405c-b381-128a788e2fb1-kube-api-access-hdcd4\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df435d78-d969-405c-b381-128a788e2fb1-config-out\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-web-config\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.606432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.606412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707064 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707064 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707305 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df435d78-d969-405c-b381-128a788e2fb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707305 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707305 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707456 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df435d78-d969-405c-b381-128a788e2fb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707510 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df435d78-d969-405c-b381-128a788e2fb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707510 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcd4\" (UniqueName: \"kubernetes.io/projected/df435d78-d969-405c-b381-128a788e2fb1-kube-api-access-hdcd4\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707616 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707616 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df435d78-d969-405c-b381-128a788e2fb1-config-out\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.707616 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-web-config\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.708166 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.708166 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.707693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df435d78-d969-405c-b381-128a788e2fb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.708166 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.708054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df435d78-d969-405c-b381-128a788e2fb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.708543 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.708515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df435d78-d969-405c-b381-128a788e2fb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.708772 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.708751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df435d78-d969-405c-b381-128a788e2fb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.710538 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.710512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.710694 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.710653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.710779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.710728 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.711003 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.710976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.711188 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.711161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df435d78-d969-405c-b381-128a788e2fb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.711188 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.711178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df435d78-d969-405c-b381-128a788e2fb1-config-out\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.711312 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.711201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.711444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.711429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.711918 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.711901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df435d78-d969-405c-b381-128a788e2fb1-web-config\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.716395 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.716340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcd4\" (UniqueName: \"kubernetes.io/projected/df435d78-d969-405c-b381-128a788e2fb1-kube-api-access-hdcd4\") pod \"alertmanager-main-0\" (UID: \"df435d78-d969-405c-b381-128a788e2fb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:38.865111 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:38.865071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:37:39.004548 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:39.004471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:37:39.007130 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:37:39.007106 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf435d78_d969_405c_b381_128a788e2fb1.slice/crio-30670ff794ffdb889682b49fafbdc9bd1e4d58f76e38b895f7c36a9176d7bb8b WatchSource:0}: Error finding container 30670ff794ffdb889682b49fafbdc9bd1e4d58f76e38b895f7c36a9176d7bb8b: Status 404 returned error can't find the container with id 30670ff794ffdb889682b49fafbdc9bd1e4d58f76e38b895f7c36a9176d7bb8b Jul 16 17:37:39.497683 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:39.497651 2564 generic.go:356] "Generic (PLEG): container finished" podID="df435d78-d969-405c-b381-128a788e2fb1" containerID="eaf49ecf8140e4a3b5062f139891ae7a1cb79c9974c119b5519444cdf98f6f61" exitCode=0 Jul 16 17:37:39.498146 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:39.497741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerDied","Data":"eaf49ecf8140e4a3b5062f139891ae7a1cb79c9974c119b5519444cdf98f6f61"} Jul 16 17:37:39.498146 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:39.497810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerStarted","Data":"30670ff794ffdb889682b49fafbdc9bd1e4d58f76e38b895f7c36a9176d7bb8b"} Jul 16 17:37:40.433405 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.433369 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5db58cfb4c-zddth"] Jul 16 17:37:40.435597 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.435581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.438166 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.438138 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 17:37:40.438398 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.438379 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 17:37:40.438581 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.438563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 17:37:40.438782 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.438767 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-smcnb\"" Jul 16 17:37:40.438990 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.438974 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 17:37:40.439201 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.439184 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 17:37:40.444968 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.444943 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 17:37:40.449306 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.449277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db58cfb4c-zddth"] Jul 16 17:37:40.506193 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.506159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerStarted","Data":"b621285536f4600351cf1297a384c79e2952799e773fb95fa1790bad13584135"} Jul 16 17:37:40.506570 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.506249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerStarted","Data":"f8a00697e3ca41481a0ed1873920341802ebeb6ee1a00978bc0c411efa3b6507"} Jul 16 17:37:40.506570 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.506279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerStarted","Data":"a71f1dc700bc10b759042cac8432e7bf0db2635a6bae9c51eb602fabe7983f4e"} Jul 16 17:37:40.506570 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.506303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerStarted","Data":"8f160be28478cea5fd8998dcbe7cae0a6e9da41095b2c472464544809ca504cc"} Jul 16 17:37:40.506570 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.506325 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerStarted","Data":"b54274006940375dae40feb7856273c538072282446876e679c47610787780d9"} Jul 16 17:37:40.506570 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.506346 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df435d78-d969-405c-b381-128a788e2fb1","Type":"ContainerStarted","Data":"a3910870bfac2fda37ebae7e94efbb8b4f9d8a01b5c79ed1a41f325dcce47f13"} Jul 16 17:37:40.521193 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.521193 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-telemeter-client-tls\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.521407 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-secret-telemeter-client\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.521407 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.521407 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjr2\" (UniqueName: \"kubernetes.io/projected/a2deeee9-5fc8-4247-9a8f-af5356184fa9-kube-api-access-xfjr2\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.521407 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-federate-client-tls\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.521564 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-serving-certs-ca-bundle\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.521564 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.521448 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-metrics-client-ca\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.533301 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.533249 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.513706516 podStartE2EDuration="2.533232526s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.52601ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:37:40.530321659 +0000 UTC m=+238.633425502" watchObservedRunningTime="2026-07-16 17:37:40.533232526 +0000 UTC m=+238.636336366" Jul 16 17:37:40.621906 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.621862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-telemeter-client-tls\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.622123 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.621930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-secret-telemeter-client\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.622123 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.622015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.622331 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.622308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjr2\" (UniqueName: \"kubernetes.io/projected/a2deeee9-5fc8-4247-9a8f-af5356184fa9-kube-api-access-xfjr2\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.622413 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.622393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-federate-client-tls\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.622472 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.622444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-serving-certs-ca-bundle\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.622522 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.622495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-metrics-client-ca\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.622582 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.622564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.623377 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.623344 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-metrics-client-ca\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.623962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.623937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.624295 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.624165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2deeee9-5fc8-4247-9a8f-af5356184fa9-serving-certs-ca-bundle\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.624872 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.624841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-secret-telemeter-client\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.625068 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.625048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.625135 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.625081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-telemeter-client-tls\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.625135 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.625124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2deeee9-5fc8-4247-9a8f-af5356184fa9-federate-client-tls\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.632503 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.632470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjr2\" (UniqueName: \"kubernetes.io/projected/a2deeee9-5fc8-4247-9a8f-af5356184fa9-kube-api-access-xfjr2\") pod \"telemeter-client-5db58cfb4c-zddth\" (UID: \"a2deeee9-5fc8-4247-9a8f-af5356184fa9\") " pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.747091 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.747004 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" Jul 16 17:37:40.907313 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:40.907274 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db58cfb4c-zddth"] Jul 16 17:37:40.910291 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:37:40.910265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2deeee9_5fc8_4247_9a8f_af5356184fa9.slice/crio-1eb0e577af384521cb2ca5e6e767f99e7a41079db6e4e4558658655611d579d4 WatchSource:0}: Error finding container 1eb0e577af384521cb2ca5e6e767f99e7a41079db6e4e4558658655611d579d4: Status 404 returned error can't find the container with id 1eb0e577af384521cb2ca5e6e767f99e7a41079db6e4e4558658655611d579d4 Jul 16 17:37:41.511491 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:41.511461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" event={"ID":"a2deeee9-5fc8-4247-9a8f-af5356184fa9","Type":"ContainerStarted","Data":"1eb0e577af384521cb2ca5e6e767f99e7a41079db6e4e4558658655611d579d4"} Jul 16 17:37:43.522334 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:43.522290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" event={"ID":"a2deeee9-5fc8-4247-9a8f-af5356184fa9","Type":"ContainerStarted","Data":"bf259fcf621404fab335e89d1ccdeb7e1f184c7af1a24e293bf9b2a4e5bdd325"} Jul 16 17:37:43.522720 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:43.522358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" event={"ID":"a2deeee9-5fc8-4247-9a8f-af5356184fa9","Type":"ContainerStarted","Data":"f61a287c8a3e289bd8d2a43dda73f6a8740714fc5cf9993b44cf681d15307ff2"} Jul 16 17:37:43.522720 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:43.522383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" event={"ID":"a2deeee9-5fc8-4247-9a8f-af5356184fa9","Type":"ContainerStarted","Data":"e6fdd83359a83d4fde061c62c97d1f1ea98753cb35cb71e368a7c9d1512dfe8b"} Jul 16 17:37:43.546504 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:43.546455 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5db58cfb4c-zddth" podStartSLOduration=1.517582075 podStartE2EDuration="3.546439906s" totalImagesPullingTime="2.028857831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:37:43.544238738 +0000 UTC m=+241.647342578" watchObservedRunningTime="2026-07-16 17:37:43.546439906 +0000 UTC m=+241.649543771" Jul 16 17:37:44.480372 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.480323 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f77d65d4-rqnjn"] Jul 16 17:37:44.483236 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.483213 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.511639 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.511592 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f77d65d4-rqnjn"] Jul 16 17:37:44.559987 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.559948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-oauth-config\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.560394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.559998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-trusted-ca-bundle\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.560394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.560022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bph\" (UniqueName: \"kubernetes.io/projected/5bc69a3b-261c-4772-901a-469321f1b568-kube-api-access-94bph\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.560394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.560062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-service-ca\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.560394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.560090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-oauth-serving-cert\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.560394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.560135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-serving-cert\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.560394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.560189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-console-config\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.661056 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.661013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-oauth-config\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.661255 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.661133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-trusted-ca-bundle\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.661255 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.661185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bph\" (UniqueName: \"kubernetes.io/projected/5bc69a3b-261c-4772-901a-469321f1b568-kube-api-access-94bph\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.661255 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.661222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-service-ca\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.661424 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.661294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-oauth-serving-cert\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.661424 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.661324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-serving-cert\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.661581 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.661561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-console-config\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.662179 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.662154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-oauth-serving-cert\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.662466 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.662444 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-trusted-ca-bundle\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.662524 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.662457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-service-ca\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.662583 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.662543 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-console-config\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.663840 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.663803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-oauth-config\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.663937 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.663923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-serving-cert\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.669643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.669619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bph\" (UniqueName: \"kubernetes.io/projected/5bc69a3b-261c-4772-901a-469321f1b568-kube-api-access-94bph\") pod \"console-79f77d65d4-rqnjn\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.793500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.793466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:44.972464 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:44.972372 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f77d65d4-rqnjn"] Jul 16 17:37:44.975129 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:37:44.975090 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc69a3b_261c_4772_901a_469321f1b568.slice/crio-e7330abf411aad251916dab8b82cc8d1412ae2deb65ba3699150f1e4342c3540 WatchSource:0}: Error finding container e7330abf411aad251916dab8b82cc8d1412ae2deb65ba3699150f1e4342c3540: Status 404 returned error can't find the container with id e7330abf411aad251916dab8b82cc8d1412ae2deb65ba3699150f1e4342c3540 Jul 16 17:37:45.534192 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:45.534158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f77d65d4-rqnjn" event={"ID":"5bc69a3b-261c-4772-901a-469321f1b568","Type":"ContainerStarted","Data":"1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e"} Jul 16 17:37:45.534360 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:45.534212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f77d65d4-rqnjn" event={"ID":"5bc69a3b-261c-4772-901a-469321f1b568","Type":"ContainerStarted","Data":"e7330abf411aad251916dab8b82cc8d1412ae2deb65ba3699150f1e4342c3540"} Jul 16 17:37:45.553963 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:45.553914 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79f77d65d4-rqnjn" podStartSLOduration=1.553896422 podStartE2EDuration="1.553896422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:37:45.552958813 +0000 UTC m=+243.656062650" watchObservedRunningTime="2026-07-16 17:37:45.553896422 +0000 UTC m=+243.657000261" Jul 16 17:37:54.793970 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:54.793936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:54.794351 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:54.793993 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:54.798712 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:54.798686 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:55.578163 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:55.578133 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:37:55.628140 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:37:55.628102 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79784fc5fb-xfhml"] Jul 16 17:38:20.650235 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.650149 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79784fc5fb-xfhml" podUID="463a87cc-aeb4-426e-9842-d3b756000fef" containerName="console" containerID="cri-o://00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078" gracePeriod=15 Jul 16 17:38:20.900802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.900750 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:38:20.979229 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979196 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-oauth-serving-cert\") pod \"463a87cc-aeb4-426e-9842-d3b756000fef\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " Jul 16 17:38:20.979418 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-oauth-config\") pod \"463a87cc-aeb4-426e-9842-d3b756000fef\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " Jul 16 17:38:20.979418 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979302 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf85b\" (UniqueName: \"kubernetes.io/projected/463a87cc-aeb4-426e-9842-d3b756000fef-kube-api-access-qf85b\") pod \"463a87cc-aeb4-426e-9842-d3b756000fef\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " Jul 16 17:38:20.979418 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979371 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-service-ca\") pod \"463a87cc-aeb4-426e-9842-d3b756000fef\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " Jul 16 17:38:20.979578 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-serving-cert\") pod \"463a87cc-aeb4-426e-9842-d3b756000fef\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " Jul 16 17:38:20.979578 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979544 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-console-config\") pod \"463a87cc-aeb4-426e-9842-d3b756000fef\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " Jul 16 17:38:20.979682 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979604 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-trusted-ca-bundle\") pod \"463a87cc-aeb4-426e-9842-d3b756000fef\" (UID: \"463a87cc-aeb4-426e-9842-d3b756000fef\") " Jul 16 17:38:20.979737 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979702 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "463a87cc-aeb4-426e-9842-d3b756000fef" (UID: "463a87cc-aeb4-426e-9842-d3b756000fef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:38:20.980046 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.979991 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-oauth-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:38:20.980314 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.980286 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-service-ca" (OuterVolumeSpecName: "service-ca") pod "463a87cc-aeb4-426e-9842-d3b756000fef" (UID: "463a87cc-aeb4-426e-9842-d3b756000fef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:38:20.980314 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.980303 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-console-config" (OuterVolumeSpecName: "console-config") pod "463a87cc-aeb4-426e-9842-d3b756000fef" (UID: "463a87cc-aeb4-426e-9842-d3b756000fef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:38:20.980463 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.980430 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "463a87cc-aeb4-426e-9842-d3b756000fef" (UID: "463a87cc-aeb4-426e-9842-d3b756000fef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:38:20.981700 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.981662 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463a87cc-aeb4-426e-9842-d3b756000fef-kube-api-access-qf85b" (OuterVolumeSpecName: "kube-api-access-qf85b") pod "463a87cc-aeb4-426e-9842-d3b756000fef" (UID: "463a87cc-aeb4-426e-9842-d3b756000fef"). InnerVolumeSpecName "kube-api-access-qf85b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:38:20.981853 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.981801 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "463a87cc-aeb4-426e-9842-d3b756000fef" (UID: "463a87cc-aeb4-426e-9842-d3b756000fef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:38:20.982411 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:20.982392 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "463a87cc-aeb4-426e-9842-d3b756000fef" (UID: "463a87cc-aeb4-426e-9842-d3b756000fef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:38:21.080408 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.080373 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:38:21.080408 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.080406 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-console-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:38:21.080408 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.080416 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-trusted-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:38:21.080640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.080424 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/463a87cc-aeb4-426e-9842-d3b756000fef-console-oauth-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:38:21.080640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.080433 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf85b\" (UniqueName: \"kubernetes.io/projected/463a87cc-aeb4-426e-9842-d3b756000fef-kube-api-access-qf85b\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:38:21.080640 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.080441 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/463a87cc-aeb4-426e-9842-d3b756000fef-service-ca\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:38:21.674288 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.674252 2564 generic.go:356] "Generic (PLEG): container finished" podID="463a87cc-aeb4-426e-9842-d3b756000fef" containerID="00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078" exitCode=0 Jul 16 17:38:21.674661 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.674311 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79784fc5fb-xfhml" Jul 16 17:38:21.674661 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.674336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79784fc5fb-xfhml" event={"ID":"463a87cc-aeb4-426e-9842-d3b756000fef","Type":"ContainerDied","Data":"00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078"} Jul 16 17:38:21.674661 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.674389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79784fc5fb-xfhml" event={"ID":"463a87cc-aeb4-426e-9842-d3b756000fef","Type":"ContainerDied","Data":"4d3721fe147010ca20fe6cd371d58f37d50c6e8ba8956d1ec052d758257387c4"} Jul 16 17:38:21.674661 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.674413 2564 scope.go:117] "RemoveContainer" containerID="00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078" Jul 16 17:38:21.683660 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.683643 2564 scope.go:117] "RemoveContainer" containerID="00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078" Jul 16 17:38:21.683934 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:38:21.683912 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078\": container with ID starting with 00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078 not found: ID does not exist" containerID="00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078" Jul 16 17:38:21.684027 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.683941 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078"} err="failed to get container status \"00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078\": rpc error: code = NotFound desc = could not find container \"00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078\": container with ID starting with 00a73ef73979ed3b2df0d84d796de0436f098b882c15e7e7e60993e2723b0078 not found: ID does not exist" Jul 16 17:38:21.696394 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.696372 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79784fc5fb-xfhml"] Jul 16 17:38:21.699910 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:21.699889 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79784fc5fb-xfhml"] Jul 16 17:38:22.573425 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:22.573394 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="463a87cc-aeb4-426e-9842-d3b756000fef" path="/var/lib/kubelet/pods/463a87cc-aeb4-426e-9842-d3b756000fef/volumes" Jul 16 17:38:42.313732 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:42.313698 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:38:42.315363 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:42.315334 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:38:42.320438 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:42.320417 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 17:38:56.480163 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.480132 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86658484bf-sh2n5"] Jul 16 17:38:56.482656 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.480570 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463a87cc-aeb4-426e-9842-d3b756000fef" containerName="console" Jul 16 17:38:56.482656 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.480583 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="463a87cc-aeb4-426e-9842-d3b756000fef" containerName="console" Jul 16 17:38:56.482656 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.480664 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="463a87cc-aeb4-426e-9842-d3b756000fef" containerName="console" Jul 16 17:38:56.483557 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.483539 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.495730 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.495709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86658484bf-sh2n5"] Jul 16 17:38:56.588210 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.588185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-serving-cert\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.588378 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.588236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-service-ca\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.588378 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.588255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-oauth-config\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.588378 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.588272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-trusted-ca-bundle\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.588378 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.588342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-config\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.588378 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.588367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ngps\" (UniqueName: \"kubernetes.io/projected/dba5ddab-d31d-4e53-acce-2b31d1b95c00-kube-api-access-4ngps\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.588557 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.588405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-oauth-serving-cert\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689141 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-config\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689272 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ngps\" (UniqueName: \"kubernetes.io/projected/dba5ddab-d31d-4e53-acce-2b31d1b95c00-kube-api-access-4ngps\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689272 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-oauth-serving-cert\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689272 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-serving-cert\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689413 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-service-ca\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689466 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-oauth-config\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689466 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-trusted-ca-bundle\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689890 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-config\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.689890 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.689883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-oauth-serving-cert\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.690046 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.690000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-service-ca\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.690244 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.690223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-trusted-ca-bundle\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.691808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.691779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-oauth-config\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.691808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.691797 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-serving-cert\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.698096 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.698054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ngps\" (UniqueName: \"kubernetes.io/projected/dba5ddab-d31d-4e53-acce-2b31d1b95c00-kube-api-access-4ngps\") pod \"console-86658484bf-sh2n5\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.792035 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.792007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:38:56.917443 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:56.917419 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86658484bf-sh2n5"] Jul 16 17:38:56.919611 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:38:56.919584 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba5ddab_d31d_4e53_acce_2b31d1b95c00.slice/crio-1c786ffdf9c1fd9a6d4b501ac471e7e04f12166c25fc955e37f80f9e51633993 WatchSource:0}: Error finding container 1c786ffdf9c1fd9a6d4b501ac471e7e04f12166c25fc955e37f80f9e51633993: Status 404 returned error can't find the container with id 1c786ffdf9c1fd9a6d4b501ac471e7e04f12166c25fc955e37f80f9e51633993 Jul 16 17:38:57.818454 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:57.818422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86658484bf-sh2n5" event={"ID":"dba5ddab-d31d-4e53-acce-2b31d1b95c00","Type":"ContainerStarted","Data":"501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9"} Jul 16 17:38:57.818808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:57.818470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86658484bf-sh2n5" event={"ID":"dba5ddab-d31d-4e53-acce-2b31d1b95c00","Type":"ContainerStarted","Data":"1c786ffdf9c1fd9a6d4b501ac471e7e04f12166c25fc955e37f80f9e51633993"} Jul 16 17:38:57.837532 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:38:57.837484 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86658484bf-sh2n5" podStartSLOduration=1.837469102 podStartE2EDuration="1.837469102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:57.835882666 +0000 UTC m=+315.938986508" watchObservedRunningTime="2026-07-16 17:38:57.837469102 +0000 UTC m=+315.940572942" Jul 16 17:39:06.793061 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:06.793026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:39:06.793409 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:06.793082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:39:06.797502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:06.797481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:39:06.859204 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:06.859183 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:39:06.910646 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:06.910618 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f77d65d4-rqnjn"] Jul 16 17:39:31.933038 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:31.932970 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79f77d65d4-rqnjn" podUID="5bc69a3b-261c-4772-901a-469321f1b568" containerName="console" containerID="cri-o://1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e" gracePeriod=15 Jul 16 17:39:32.176529 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.176499 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:39:32.282073 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282040 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-trusted-ca-bundle\") pod \"5bc69a3b-261c-4772-901a-469321f1b568\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " Jul 16 17:39:32.282251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282086 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-oauth-serving-cert\") pod \"5bc69a3b-261c-4772-901a-469321f1b568\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " Jul 16 17:39:32.282251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282118 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-oauth-config\") pod \"5bc69a3b-261c-4772-901a-469321f1b568\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " Jul 16 17:39:32.282251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282137 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-console-config\") pod \"5bc69a3b-261c-4772-901a-469321f1b568\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " Jul 16 17:39:32.282251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282155 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-service-ca\") pod \"5bc69a3b-261c-4772-901a-469321f1b568\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " Jul 16 17:39:32.282251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282194 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94bph\" (UniqueName: \"kubernetes.io/projected/5bc69a3b-261c-4772-901a-469321f1b568-kube-api-access-94bph\") pod \"5bc69a3b-261c-4772-901a-469321f1b568\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " Jul 16 17:39:32.282251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282217 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-serving-cert\") pod \"5bc69a3b-261c-4772-901a-469321f1b568\" (UID: \"5bc69a3b-261c-4772-901a-469321f1b568\") " Jul 16 17:39:32.282543 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282472 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5bc69a3b-261c-4772-901a-469321f1b568" (UID: "5bc69a3b-261c-4772-901a-469321f1b568"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:39:32.282752 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282721 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5bc69a3b-261c-4772-901a-469321f1b568" (UID: "5bc69a3b-261c-4772-901a-469321f1b568"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:39:32.282913 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282759 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-service-ca" (OuterVolumeSpecName: "service-ca") pod "5bc69a3b-261c-4772-901a-469321f1b568" (UID: "5bc69a3b-261c-4772-901a-469321f1b568"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:39:32.282913 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.282766 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-console-config" (OuterVolumeSpecName: "console-config") pod "5bc69a3b-261c-4772-901a-469321f1b568" (UID: "5bc69a3b-261c-4772-901a-469321f1b568"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:39:32.284392 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.284362 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5bc69a3b-261c-4772-901a-469321f1b568" (UID: "5bc69a3b-261c-4772-901a-469321f1b568"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:39:32.284392 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.284381 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc69a3b-261c-4772-901a-469321f1b568-kube-api-access-94bph" (OuterVolumeSpecName: "kube-api-access-94bph") pod "5bc69a3b-261c-4772-901a-469321f1b568" (UID: "5bc69a3b-261c-4772-901a-469321f1b568"). InnerVolumeSpecName "kube-api-access-94bph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:39:32.284515 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.284452 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5bc69a3b-261c-4772-901a-469321f1b568" (UID: "5bc69a3b-261c-4772-901a-469321f1b568"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:39:32.383066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.383008 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-oauth-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:39:32.383066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.383057 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-oauth-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:39:32.383066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.383068 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-console-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:39:32.383066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.383077 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-service-ca\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:39:32.383066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.383085 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bph\" (UniqueName: \"kubernetes.io/projected/5bc69a3b-261c-4772-901a-469321f1b568-kube-api-access-94bph\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:39:32.383375 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.383094 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bc69a3b-261c-4772-901a-469321f1b568-console-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:39:32.383375 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.383103 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bc69a3b-261c-4772-901a-469321f1b568-trusted-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:39:32.958668 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.958574 2564 generic.go:356] "Generic (PLEG): container finished" podID="5bc69a3b-261c-4772-901a-469321f1b568" containerID="1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e" exitCode=0 Jul 16 17:39:32.958668 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.958624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f77d65d4-rqnjn" event={"ID":"5bc69a3b-261c-4772-901a-469321f1b568","Type":"ContainerDied","Data":"1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e"} Jul 16 17:39:32.958668 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.958626 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f77d65d4-rqnjn" Jul 16 17:39:32.958668 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.958664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f77d65d4-rqnjn" event={"ID":"5bc69a3b-261c-4772-901a-469321f1b568","Type":"ContainerDied","Data":"e7330abf411aad251916dab8b82cc8d1412ae2deb65ba3699150f1e4342c3540"} Jul 16 17:39:32.959288 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.958683 2564 scope.go:117] "RemoveContainer" containerID="1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e" Jul 16 17:39:32.966781 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.966763 2564 scope.go:117] "RemoveContainer" containerID="1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e" Jul 16 17:39:32.967081 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:39:32.967062 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e\": container with ID starting with 1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e not found: ID does not exist" containerID="1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e" Jul 16 17:39:32.967133 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.967089 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e"} err="failed to get container status \"1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e\": rpc error: code = NotFound desc = could not find container \"1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e\": container with ID starting with 1be3be82f070596ab5dfc5eef9bc2330be32516946ca6229d0704f395215556e not found: ID does not exist" Jul 16 17:39:32.976452 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.976419 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f77d65d4-rqnjn"] Jul 16 17:39:32.978251 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:32.978224 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f77d65d4-rqnjn"] Jul 16 17:39:34.574107 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:39:34.574031 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bc69a3b-261c-4772-901a-469321f1b568" path="/var/lib/kubelet/pods/5bc69a3b-261c-4772-901a-469321f1b568/volumes" Jul 16 17:40:37.430165 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.430126 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg"] Jul 16 17:40:37.430615 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.430481 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc69a3b-261c-4772-901a-469321f1b568" containerName="console" Jul 16 17:40:37.430615 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.430494 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc69a3b-261c-4772-901a-469321f1b568" containerName="console" Jul 16 17:40:37.430615 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.430551 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc69a3b-261c-4772-901a-469321f1b568" containerName="console" Jul 16 17:40:37.433456 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.433439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.436145 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.436119 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 17:40:37.436856 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.436819 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 17:40:37.436956 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.436819 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dhcbn\"" Jul 16 17:40:37.442634 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.442609 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg"] Jul 16 17:40:37.576718 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.576694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.576862 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.576731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjvr\" (UniqueName: \"kubernetes.io/projected/372e4886-b065-4416-ae3d-3f50502022a5-kube-api-access-8wjvr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.576862 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.576783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.677279 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.677245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.677405 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.677295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjvr\" (UniqueName: \"kubernetes.io/projected/372e4886-b065-4416-ae3d-3f50502022a5-kube-api-access-8wjvr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.677405 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.677364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.677630 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.677610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.677716 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.677694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.687028 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.686977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjvr\" (UniqueName: \"kubernetes.io/projected/372e4886-b065-4416-ae3d-3f50502022a5-kube-api-access-8wjvr\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.744058 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.744031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:37.867545 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.867518 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg"] Jul 16 17:40:37.870520 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:40:37.870486 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372e4886_b065_4416_ae3d_3f50502022a5.slice/crio-a25c861b0212ac6de538fb5b11fce287341d764a354097b71324a9ebe5d0ceb6 WatchSource:0}: Error finding container a25c861b0212ac6de538fb5b11fce287341d764a354097b71324a9ebe5d0ceb6: Status 404 returned error can't find the container with id a25c861b0212ac6de538fb5b11fce287341d764a354097b71324a9ebe5d0ceb6 Jul 16 17:40:37.872647 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:37.872630 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:40:38.223726 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:38.223695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" event={"ID":"372e4886-b065-4416-ae3d-3f50502022a5","Type":"ContainerStarted","Data":"a25c861b0212ac6de538fb5b11fce287341d764a354097b71324a9ebe5d0ceb6"} Jul 16 17:40:46.261763 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:46.261725 2564 generic.go:356] "Generic (PLEG): container finished" podID="372e4886-b065-4416-ae3d-3f50502022a5" containerID="b7d88cc3645987e2a0e3805bd84f7e0dbaa674d88445b2a3e08fa3f3f6d11c6d" exitCode=0 Jul 16 17:40:46.262243 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:46.261842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" event={"ID":"372e4886-b065-4416-ae3d-3f50502022a5","Type":"ContainerDied","Data":"b7d88cc3645987e2a0e3805bd84f7e0dbaa674d88445b2a3e08fa3f3f6d11c6d"} Jul 16 17:40:49.275997 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:49.275965 2564 generic.go:356] "Generic (PLEG): container finished" podID="372e4886-b065-4416-ae3d-3f50502022a5" containerID="4ad67a23bf0f6240520a8dc1935d5279f0d2b8e1c6d8be4c2715efb2810791ce" exitCode=0 Jul 16 17:40:49.276456 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:49.276048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" event={"ID":"372e4886-b065-4416-ae3d-3f50502022a5","Type":"ContainerDied","Data":"4ad67a23bf0f6240520a8dc1935d5279f0d2b8e1c6d8be4c2715efb2810791ce"} Jul 16 17:40:56.309151 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:56.309117 2564 generic.go:356] "Generic (PLEG): container finished" podID="372e4886-b065-4416-ae3d-3f50502022a5" containerID="5bbf2d3152266092e79939e34f2cc66f1584e85c26e9d8178e7b076bfad85032" exitCode=0 Jul 16 17:40:56.309620 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:56.309182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" event={"ID":"372e4886-b065-4416-ae3d-3f50502022a5","Type":"ContainerDied","Data":"5bbf2d3152266092e79939e34f2cc66f1584e85c26e9d8178e7b076bfad85032"} Jul 16 17:40:57.439975 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.439950 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:57.440600 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.440579 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wjvr\" (UniqueName: \"kubernetes.io/projected/372e4886-b065-4416-ae3d-3f50502022a5-kube-api-access-8wjvr\") pod \"372e4886-b065-4416-ae3d-3f50502022a5\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " Jul 16 17:40:57.440703 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.440640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-util\") pod \"372e4886-b065-4416-ae3d-3f50502022a5\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " Jul 16 17:40:57.440703 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.440676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-bundle\") pod \"372e4886-b065-4416-ae3d-3f50502022a5\" (UID: \"372e4886-b065-4416-ae3d-3f50502022a5\") " Jul 16 17:40:57.441306 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.441280 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-bundle" (OuterVolumeSpecName: "bundle") pod "372e4886-b065-4416-ae3d-3f50502022a5" (UID: "372e4886-b065-4416-ae3d-3f50502022a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:40:57.442697 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.442673 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372e4886-b065-4416-ae3d-3f50502022a5-kube-api-access-8wjvr" (OuterVolumeSpecName: "kube-api-access-8wjvr") pod "372e4886-b065-4416-ae3d-3f50502022a5" (UID: "372e4886-b065-4416-ae3d-3f50502022a5"). InnerVolumeSpecName "kube-api-access-8wjvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:40:57.446589 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.446558 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-util" (OuterVolumeSpecName: "util") pod "372e4886-b065-4416-ae3d-3f50502022a5" (UID: "372e4886-b065-4416-ae3d-3f50502022a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:40:57.541384 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.541351 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjvr\" (UniqueName: \"kubernetes.io/projected/372e4886-b065-4416-ae3d-3f50502022a5-kube-api-access-8wjvr\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:40:57.541384 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.541378 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-util\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:40:57.541384 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:57.541390 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/372e4886-b065-4416-ae3d-3f50502022a5-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:40:58.318525 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:58.318495 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" Jul 16 17:40:58.318525 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:58.318517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5s82sg" event={"ID":"372e4886-b065-4416-ae3d-3f50502022a5","Type":"ContainerDied","Data":"a25c861b0212ac6de538fb5b11fce287341d764a354097b71324a9ebe5d0ceb6"} Jul 16 17:40:58.318729 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:40:58.318558 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a25c861b0212ac6de538fb5b11fce287341d764a354097b71324a9ebe5d0ceb6" Jul 16 17:41:05.140193 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140111 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx"] Jul 16 17:41:05.140617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140413 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372e4886-b065-4416-ae3d-3f50502022a5" containerName="pull" Jul 16 17:41:05.140617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140423 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="372e4886-b065-4416-ae3d-3f50502022a5" containerName="pull" Jul 16 17:41:05.140617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140432 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372e4886-b065-4416-ae3d-3f50502022a5" containerName="extract" Jul 16 17:41:05.140617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140437 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="372e4886-b065-4416-ae3d-3f50502022a5" containerName="extract" Jul 16 17:41:05.140617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140449 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372e4886-b065-4416-ae3d-3f50502022a5" containerName="util" Jul 16 17:41:05.140617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140454 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="372e4886-b065-4416-ae3d-3f50502022a5" containerName="util" Jul 16 17:41:05.140617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.140502 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="372e4886-b065-4416-ae3d-3f50502022a5" containerName="extract" Jul 16 17:41:05.143644 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.143624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.146005 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.145984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 16 17:41:05.146117 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.146040 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 16 17:41:05.146117 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.146066 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 16 17:41:05.146226 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.146040 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-52dbr\"" Jul 16 17:41:05.160496 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.160474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx"] Jul 16 17:41:05.192257 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.192230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/000875d4-09be-48d4-96f4-3a1909c19f67-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx\" (UID: \"000875d4-09be-48d4-96f4-3a1909c19f67\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.192378 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.192334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltcx\" (UniqueName: \"kubernetes.io/projected/000875d4-09be-48d4-96f4-3a1909c19f67-kube-api-access-9ltcx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx\" (UID: \"000875d4-09be-48d4-96f4-3a1909c19f67\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.293528 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.293501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/000875d4-09be-48d4-96f4-3a1909c19f67-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx\" (UID: \"000875d4-09be-48d4-96f4-3a1909c19f67\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.293637 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.293599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltcx\" (UniqueName: \"kubernetes.io/projected/000875d4-09be-48d4-96f4-3a1909c19f67-kube-api-access-9ltcx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx\" (UID: \"000875d4-09be-48d4-96f4-3a1909c19f67\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.295815 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.295793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/000875d4-09be-48d4-96f4-3a1909c19f67-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx\" (UID: \"000875d4-09be-48d4-96f4-3a1909c19f67\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.303360 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.303337 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltcx\" (UniqueName: \"kubernetes.io/projected/000875d4-09be-48d4-96f4-3a1909c19f67-kube-api-access-9ltcx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx\" (UID: \"000875d4-09be-48d4-96f4-3a1909c19f67\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.453173 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.453109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:05.581750 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:05.581719 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx"] Jul 16 17:41:05.585007 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:41:05.584978 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000875d4_09be_48d4_96f4_3a1909c19f67.slice/crio-64c53c126e90e5e29c4173a05e942a83bfa72f73d11ed94c5553b2704984b665 WatchSource:0}: Error finding container 64c53c126e90e5e29c4173a05e942a83bfa72f73d11ed94c5553b2704984b665: Status 404 returned error can't find the container with id 64c53c126e90e5e29c4173a05e942a83bfa72f73d11ed94c5553b2704984b665 Jul 16 17:41:06.350182 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:06.350137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" event={"ID":"000875d4-09be-48d4-96f4-3a1909c19f67","Type":"ContainerStarted","Data":"64c53c126e90e5e29c4173a05e942a83bfa72f73d11ed94c5553b2704984b665"} Jul 16 17:41:10.118176 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.118143 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-m9cmf"] Jul 16 17:41:10.120374 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.120349 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.122744 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.122710 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 16 17:41:10.122891 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.122747 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 16 17:41:10.122891 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.122793 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xx5gn\"" Jul 16 17:41:10.131632 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.131604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-certificates\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.131744 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.131670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3d630619-8677-4793-8048-f8430a83972d-cabundle0\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.131794 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.131771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7xh\" (UniqueName: \"kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-kube-api-access-px7xh\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.131948 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.131934 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-m9cmf"] Jul 16 17:41:10.232444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.232413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-certificates\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.232638 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.232459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3d630619-8677-4793-8048-f8430a83972d-cabundle0\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.232638 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.232582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7xh\" (UniqueName: \"kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-kube-api-access-px7xh\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.232755 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.232585 2564 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Jul 16 17:41:10.232755 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.232677 2564 secret.go:281] references non-existent secret key: ca.crt Jul 16 17:41:10.232755 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.232690 2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 16 17:41:10.232755 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.232727 2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-m9cmf: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 16 17:41:10.232950 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.232788 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-certificates podName:3d630619-8677-4793-8048-f8430a83972d nodeName:}" failed. No retries permitted until 2026-07-16 17:41:10.732768124 +0000 UTC m=+448.835871956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-certificates") pod "keda-operator-77f648d694-m9cmf" (UID: "3d630619-8677-4793-8048-f8430a83972d") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 16 17:41:10.233243 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.233217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3d630619-8677-4793-8048-f8430a83972d-cabundle0\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.244118 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.244093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7xh\" (UniqueName: \"kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-kube-api-access-px7xh\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.370718 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.370633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" event={"ID":"000875d4-09be-48d4-96f4-3a1909c19f67","Type":"ContainerStarted","Data":"e63574d67fb2c65cfabc61538273fce8b6e1a82e6ff193d0ce5958893caee1d6"} Jul 16 17:41:10.370894 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.370876 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:10.394946 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.394900 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" podStartSLOduration=1.542193142 podStartE2EDuration="5.394884778s" totalImagesPullingTime="3.852691636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:10.39326581 +0000 UTC m=+448.496369649" watchObservedRunningTime="2026-07-16 17:41:10.394884778 +0000 UTC m=+448.497988617" Jul 16 17:41:10.732338 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.732250 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-kgpg4"] Jul 16 17:41:10.734720 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.734698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:10.736976 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.736955 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 16 17:41:10.737203 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.737177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-certificates\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.739653 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.739630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3d630619-8677-4793-8048-f8430a83972d-certificates\") pod \"keda-operator-77f648d694-m9cmf\" (UID: \"3d630619-8677-4793-8048-f8430a83972d\") " pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:10.745430 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.745402 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-kgpg4"] Jul 16 17:41:10.838201 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.838160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scp86\" (UniqueName: \"kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-kube-api-access-scp86\") pod \"keda-admission-56f9798bb4-kgpg4\" (UID: \"10ad0f19-d92a-401c-9196-a968d3d85d88\") " pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:10.838347 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.838228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-certificates\") pod \"keda-admission-56f9798bb4-kgpg4\" (UID: \"10ad0f19-d92a-401c-9196-a968d3d85d88\") " pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:10.939637 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.939596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-certificates\") pod \"keda-admission-56f9798bb4-kgpg4\" (UID: \"10ad0f19-d92a-401c-9196-a968d3d85d88\") " pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:10.939807 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.939691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scp86\" (UniqueName: \"kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-kube-api-access-scp86\") pod \"keda-admission-56f9798bb4-kgpg4\" (UID: \"10ad0f19-d92a-401c-9196-a968d3d85d88\") " pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:10.939807 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.939761 2564 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Jul 16 17:41:10.939807 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.939785 2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-kgpg4: secret "keda-admission-webhooks-certs" not found Jul 16 17:41:10.939934 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:41:10.939879 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-certificates podName:10ad0f19-d92a-401c-9196-a968d3d85d88 nodeName:}" failed. No retries permitted until 2026-07-16 17:41:11.43986008 +0000 UTC m=+449.542963902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-certificates") pod "keda-admission-56f9798bb4-kgpg4" (UID: "10ad0f19-d92a-401c-9196-a968d3d85d88") : secret "keda-admission-webhooks-certs" not found Jul 16 17:41:10.954734 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:10.954701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scp86\" (UniqueName: \"kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-kube-api-access-scp86\") pod \"keda-admission-56f9798bb4-kgpg4\" (UID: \"10ad0f19-d92a-401c-9196-a968d3d85d88\") " pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:11.030503 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:11.030465 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:11.189386 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:11.189355 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-m9cmf"] Jul 16 17:41:11.192804 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:41:11.192772 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d630619_8677_4793_8048_f8430a83972d.slice/crio-81d77d6e0e4fecdd6e5322959a001d1b18da5c5ac1e5271adffb3594700d601b WatchSource:0}: Error finding container 81d77d6e0e4fecdd6e5322959a001d1b18da5c5ac1e5271adffb3594700d601b: Status 404 returned error can't find the container with id 81d77d6e0e4fecdd6e5322959a001d1b18da5c5ac1e5271adffb3594700d601b Jul 16 17:41:11.376962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:11.376866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-m9cmf" event={"ID":"3d630619-8677-4793-8048-f8430a83972d","Type":"ContainerStarted","Data":"81d77d6e0e4fecdd6e5322959a001d1b18da5c5ac1e5271adffb3594700d601b"} Jul 16 17:41:11.446970 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:11.446931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-certificates\") pod \"keda-admission-56f9798bb4-kgpg4\" (UID: \"10ad0f19-d92a-401c-9196-a968d3d85d88\") " pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:11.449435 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:11.449410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/10ad0f19-d92a-401c-9196-a968d3d85d88-certificates\") pod \"keda-admission-56f9798bb4-kgpg4\" (UID: \"10ad0f19-d92a-401c-9196-a968d3d85d88\") " pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:11.654467 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:11.654380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:11.791103 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:11.791078 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-kgpg4"] Jul 16 17:41:11.792708 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:41:11.792677 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ad0f19_d92a_401c_9196_a968d3d85d88.slice/crio-2cfd8135f0ac02247a81e4d88c8b5c07908928ab8093d7db0ef390e08c4d1b97 WatchSource:0}: Error finding container 2cfd8135f0ac02247a81e4d88c8b5c07908928ab8093d7db0ef390e08c4d1b97: Status 404 returned error can't find the container with id 2cfd8135f0ac02247a81e4d88c8b5c07908928ab8093d7db0ef390e08c4d1b97 Jul 16 17:41:12.383617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:12.383581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" event={"ID":"10ad0f19-d92a-401c-9196-a968d3d85d88","Type":"ContainerStarted","Data":"2cfd8135f0ac02247a81e4d88c8b5c07908928ab8093d7db0ef390e08c4d1b97"} Jul 16 17:41:13.391135 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:13.391098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" event={"ID":"10ad0f19-d92a-401c-9196-a968d3d85d88","Type":"ContainerStarted","Data":"45fe4dcea9c03309b0073afaf9f130b3a22048168e7a12e2bf262e821c1f6e47"} Jul 16 17:41:13.391575 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:13.391269 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:13.409305 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:13.409229 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" podStartSLOduration=1.936082473 podStartE2EDuration="3.409208815s" totalImagesPullingTime="1.473126342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:13.40872121 +0000 UTC m=+451.511825051" watchObservedRunningTime="2026-07-16 17:41:13.409208815 +0000 UTC m=+451.512312656" Jul 16 17:41:16.406184 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:16.406151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-m9cmf" event={"ID":"3d630619-8677-4793-8048-f8430a83972d","Type":"ContainerStarted","Data":"a1eed94e120360109d695982821b737113b5026f79024d76e0a67c147d51e847"} Jul 16 17:41:16.406562 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:16.406271 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:41:16.423699 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:16.423657 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-m9cmf" podStartSLOduration=1.613212215 podStartE2EDuration="6.423645231s" totalImagesPullingTime="4.810433016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:16.42188648 +0000 UTC m=+454.524990322" watchObservedRunningTime="2026-07-16 17:41:16.423645231 +0000 UTC m=+454.526749070" Jul 16 17:41:31.379500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:31.379467 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fr4jx" Jul 16 17:41:34.398403 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:34.398364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-kgpg4" Jul 16 17:41:37.412630 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:41:37.412592 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-m9cmf" Jul 16 17:42:17.577572 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.577539 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-24tcj"] Jul 16 17:42:17.579896 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.579878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.582131 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.582112 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 16 17:42:17.582812 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.582795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-pvv7c\"" Jul 16 17:42:17.582812 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.582804 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 16 17:42:17.582967 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.582808 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 16 17:42:17.593310 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.593289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-24tcj"] Jul 16 17:42:17.611896 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.611868 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-vvjr7"] Jul 16 17:42:17.614550 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.614532 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:17.618364 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.618336 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xvd56\"" Jul 16 17:42:17.618456 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.618341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 16 17:42:17.623315 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.623297 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vvjr7"] Jul 16 17:42:17.684712 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.684685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qb6m\" (UniqueName: \"kubernetes.io/projected/7098ba81-2f72-44d6-ae62-75d1a5fb5fbb-kube-api-access-7qb6m\") pod \"llmisvc-controller-manager-654c557856-24tcj\" (UID: \"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb\") " pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.684848 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.684716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkznx\" (UniqueName: \"kubernetes.io/projected/05f3e46f-b716-48b1-825e-6cc8c5e17516-kube-api-access-jkznx\") pod \"seaweedfs-64568bcd49-vvjr7\" (UID: \"05f3e46f-b716-48b1-825e-6cc8c5e17516\") " pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:17.684848 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.684763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05f3e46f-b716-48b1-825e-6cc8c5e17516-data\") pod \"seaweedfs-64568bcd49-vvjr7\" (UID: \"05f3e46f-b716-48b1-825e-6cc8c5e17516\") " pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:17.684848 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.684796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7098ba81-2f72-44d6-ae62-75d1a5fb5fbb-cert\") pod \"llmisvc-controller-manager-654c557856-24tcj\" (UID: \"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb\") " pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.785432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.785388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qb6m\" (UniqueName: \"kubernetes.io/projected/7098ba81-2f72-44d6-ae62-75d1a5fb5fbb-kube-api-access-7qb6m\") pod \"llmisvc-controller-manager-654c557856-24tcj\" (UID: \"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb\") " pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.785432 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.785437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkznx\" (UniqueName: \"kubernetes.io/projected/05f3e46f-b716-48b1-825e-6cc8c5e17516-kube-api-access-jkznx\") pod \"seaweedfs-64568bcd49-vvjr7\" (UID: \"05f3e46f-b716-48b1-825e-6cc8c5e17516\") " pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:17.785650 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.785515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05f3e46f-b716-48b1-825e-6cc8c5e17516-data\") pod \"seaweedfs-64568bcd49-vvjr7\" (UID: \"05f3e46f-b716-48b1-825e-6cc8c5e17516\") " pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:17.785650 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.785577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7098ba81-2f72-44d6-ae62-75d1a5fb5fbb-cert\") pod \"llmisvc-controller-manager-654c557856-24tcj\" (UID: \"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb\") " pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.785888 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.785866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05f3e46f-b716-48b1-825e-6cc8c5e17516-data\") pod \"seaweedfs-64568bcd49-vvjr7\" (UID: \"05f3e46f-b716-48b1-825e-6cc8c5e17516\") " pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:17.787897 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.787880 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7098ba81-2f72-44d6-ae62-75d1a5fb5fbb-cert\") pod \"llmisvc-controller-manager-654c557856-24tcj\" (UID: \"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb\") " pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.795090 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.795061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkznx\" (UniqueName: \"kubernetes.io/projected/05f3e46f-b716-48b1-825e-6cc8c5e17516-kube-api-access-jkznx\") pod \"seaweedfs-64568bcd49-vvjr7\" (UID: \"05f3e46f-b716-48b1-825e-6cc8c5e17516\") " pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:17.795594 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.795575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qb6m\" (UniqueName: \"kubernetes.io/projected/7098ba81-2f72-44d6-ae62-75d1a5fb5fbb-kube-api-access-7qb6m\") pod \"llmisvc-controller-manager-654c557856-24tcj\" (UID: \"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb\") " pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.889876 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.889776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:17.923818 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:17.923791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:18.026979 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:18.026949 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-24tcj"] Jul 16 17:42:18.028892 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:42:18.028862 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7098ba81_2f72_44d6_ae62_75d1a5fb5fbb.slice/crio-89f7d6116436864e15f85f907b41a412d00b038b5e9e77a1348f5edfc86b9265 WatchSource:0}: Error finding container 89f7d6116436864e15f85f907b41a412d00b038b5e9e77a1348f5edfc86b9265: Status 404 returned error can't find the container with id 89f7d6116436864e15f85f907b41a412d00b038b5e9e77a1348f5edfc86b9265 Jul 16 17:42:18.073549 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:18.073524 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vvjr7"] Jul 16 17:42:18.076029 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:42:18.075992 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f3e46f_b716_48b1_825e_6cc8c5e17516.slice/crio-43423c30236701464efb459fe9da794f591ca2699f096b25593b0a76d3f2eedc WatchSource:0}: Error finding container 43423c30236701464efb459fe9da794f591ca2699f096b25593b0a76d3f2eedc: Status 404 returned error can't find the container with id 43423c30236701464efb459fe9da794f591ca2699f096b25593b0a76d3f2eedc Jul 16 17:42:18.666082 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:18.666047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vvjr7" event={"ID":"05f3e46f-b716-48b1-825e-6cc8c5e17516","Type":"ContainerStarted","Data":"43423c30236701464efb459fe9da794f591ca2699f096b25593b0a76d3f2eedc"} Jul 16 17:42:18.667497 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:18.667435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-24tcj" event={"ID":"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb","Type":"ContainerStarted","Data":"89f7d6116436864e15f85f907b41a412d00b038b5e9e77a1348f5edfc86b9265"} Jul 16 17:42:21.683295 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:21.683256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-24tcj" event={"ID":"7098ba81-2f72-44d6-ae62-75d1a5fb5fbb","Type":"ContainerStarted","Data":"ed4fa8d2b076ee04cbf22181e8cfac97aad90163c851773d908a1d497929993f"} Jul 16 17:42:21.683734 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:21.683349 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:42:21.684709 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:21.684689 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vvjr7" event={"ID":"05f3e46f-b716-48b1-825e-6cc8c5e17516","Type":"ContainerStarted","Data":"9ef7c6ed7f6f7f8e17c8b418ca3b578d0d86bbfe073a3f39b00c8a25ff2baacb"} Jul 16 17:42:21.684875 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:21.684740 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:21.700168 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:21.700124 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-24tcj" podStartSLOduration=1.488996589 podStartE2EDuration="4.700109206s" totalImagesPullingTime="3.211112617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:42:21.699659665 +0000 UTC m=+519.802763518" watchObservedRunningTime="2026-07-16 17:42:21.700109206 +0000 UTC m=+519.803213047" Jul 16 17:42:21.714300 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:21.714261 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-vvjr7" podStartSLOduration=1.500709521 podStartE2EDuration="4.714249671s" totalImagesPullingTime="3.21354015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:42:21.71380466 +0000 UTC m=+519.816908496" watchObservedRunningTime="2026-07-16 17:42:21.714249671 +0000 UTC m=+519.817353512" Jul 16 17:42:27.690525 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:27.690498 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-vvjr7" Jul 16 17:42:51.270348 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.270319 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cdc4d869f-kllzd"] Jul 16 17:42:51.275014 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.274995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.286179 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.286160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cdc4d869f-kllzd"] Jul 16 17:42:51.355019 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.354984 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-oauth-config\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.355019 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.355022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plsfs\" (UniqueName: \"kubernetes.io/projected/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-kube-api-access-plsfs\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.355219 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.355044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-service-ca\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.355219 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.355071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-trusted-ca-bundle\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.355219 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.355113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-oauth-serving-cert\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.355219 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.355131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-config\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.355219 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.355150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-serving-cert\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.456515 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.456472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-oauth-config\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.456515 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.456517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plsfs\" (UniqueName: \"kubernetes.io/projected/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-kube-api-access-plsfs\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.456747 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.456552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-service-ca\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.456747 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.456573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-trusted-ca-bundle\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.456747 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.456604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-oauth-serving-cert\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.456747 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.456627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-config\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.456747 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.456657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-serving-cert\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.457400 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.457371 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-service-ca\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.457495 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.457404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-config\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.457495 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.457416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-oauth-serving-cert\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.457570 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.457512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-trusted-ca-bundle\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.459273 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.459254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-oauth-config\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.459468 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.459447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-console-serving-cert\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.466393 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.466369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plsfs\" (UniqueName: \"kubernetes.io/projected/b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f-kube-api-access-plsfs\") pod \"console-5cdc4d869f-kllzd\" (UID: \"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f\") " pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.583514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.583481 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:42:51.711755 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.711729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cdc4d869f-kllzd"] Jul 16 17:42:51.714286 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:42:51.714256 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f3ae94_de86_49cb_aeff_3a0e7cd9c28f.slice/crio-5e7f72e815be9e01617d69d1d81aa18a07f0697d4b6b8b8c595cc46735d82fad WatchSource:0}: Error finding container 5e7f72e815be9e01617d69d1d81aa18a07f0697d4b6b8b8c595cc46735d82fad: Status 404 returned error can't find the container with id 5e7f72e815be9e01617d69d1d81aa18a07f0697d4b6b8b8c595cc46735d82fad Jul 16 17:42:51.816812 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.816775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdc4d869f-kllzd" event={"ID":"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f","Type":"ContainerStarted","Data":"d6b21f5a08d170d5b880a2b81fb1e0cfecc6b53749e047b777da6c886e1c77f6"} Jul 16 17:42:51.817010 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.816848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdc4d869f-kllzd" event={"ID":"b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f","Type":"ContainerStarted","Data":"5e7f72e815be9e01617d69d1d81aa18a07f0697d4b6b8b8c595cc46735d82fad"} Jul 16 17:42:51.835528 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:51.835428 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cdc4d869f-kllzd" podStartSLOduration=0.83539824 podStartE2EDuration="835.39824ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:42:51.834200975 +0000 UTC m=+549.937304828" watchObservedRunningTime="2026-07-16 17:42:51.83539824 +0000 UTC m=+549.938502079" Jul 16 17:42:52.691425 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:42:52.691397 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-24tcj" Jul 16 17:43:01.584462 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:01.584419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:43:01.584901 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:01.584503 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:43:01.589304 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:01.589281 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:43:01.867301 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:01.867227 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cdc4d869f-kllzd" Jul 16 17:43:01.919895 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:01.919865 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86658484bf-sh2n5"] Jul 16 17:43:26.942713 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:26.942670 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86658484bf-sh2n5" podUID="dba5ddab-d31d-4e53-acce-2b31d1b95c00" containerName="console" containerID="cri-o://501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9" gracePeriod=15 Jul 16 17:43:27.186521 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.186498 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:43:27.352502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352477 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ngps\" (UniqueName: \"kubernetes.io/projected/dba5ddab-d31d-4e53-acce-2b31d1b95c00-kube-api-access-4ngps\") pod \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " Jul 16 17:43:27.352664 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352529 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-oauth-serving-cert\") pod \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " Jul 16 17:43:27.352664 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-trusted-ca-bundle\") pod \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " Jul 16 17:43:27.352664 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-serving-cert\") pod \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " Jul 16 17:43:27.352664 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-oauth-config\") pod \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " Jul 16 17:43:27.352664 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352638 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-service-ca\") pod \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " Jul 16 17:43:27.352664 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352660 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-config\") pod \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\" (UID: \"dba5ddab-d31d-4e53-acce-2b31d1b95c00\") " Jul 16 17:43:27.353051 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.352917 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dba5ddab-d31d-4e53-acce-2b31d1b95c00" (UID: "dba5ddab-d31d-4e53-acce-2b31d1b95c00"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:43:27.353171 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.353011 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dba5ddab-d31d-4e53-acce-2b31d1b95c00" (UID: "dba5ddab-d31d-4e53-acce-2b31d1b95c00"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:43:27.353171 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.353086 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-service-ca" (OuterVolumeSpecName: "service-ca") pod "dba5ddab-d31d-4e53-acce-2b31d1b95c00" (UID: "dba5ddab-d31d-4e53-acce-2b31d1b95c00"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:43:27.353253 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.353179 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-config" (OuterVolumeSpecName: "console-config") pod "dba5ddab-d31d-4e53-acce-2b31d1b95c00" (UID: "dba5ddab-d31d-4e53-acce-2b31d1b95c00"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:43:27.354782 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.354756 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dba5ddab-d31d-4e53-acce-2b31d1b95c00" (UID: "dba5ddab-d31d-4e53-acce-2b31d1b95c00"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:27.355248 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.355224 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dba5ddab-d31d-4e53-acce-2b31d1b95c00" (UID: "dba5ddab-d31d-4e53-acce-2b31d1b95c00"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:43:27.355248 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.355223 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba5ddab-d31d-4e53-acce-2b31d1b95c00-kube-api-access-4ngps" (OuterVolumeSpecName: "kube-api-access-4ngps") pod "dba5ddab-d31d-4e53-acce-2b31d1b95c00" (UID: "dba5ddab-d31d-4e53-acce-2b31d1b95c00"). InnerVolumeSpecName "kube-api-access-4ngps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:43:27.454124 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.454094 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-service-ca\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:43:27.454124 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.454120 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:43:27.454124 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.454129 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ngps\" (UniqueName: \"kubernetes.io/projected/dba5ddab-d31d-4e53-acce-2b31d1b95c00-kube-api-access-4ngps\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:43:27.454325 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.454138 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-oauth-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:43:27.454325 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.454147 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba5ddab-d31d-4e53-acce-2b31d1b95c00-trusted-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:43:27.454325 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.454156 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-serving-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:43:27.454325 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.454165 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba5ddab-d31d-4e53-acce-2b31d1b95c00-console-oauth-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:43:27.981921 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.981878 2564 generic.go:356] "Generic (PLEG): container finished" podID="dba5ddab-d31d-4e53-acce-2b31d1b95c00" containerID="501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9" exitCode=0 Jul 16 17:43:27.982296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.981929 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86658484bf-sh2n5" Jul 16 17:43:27.982296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.981963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86658484bf-sh2n5" event={"ID":"dba5ddab-d31d-4e53-acce-2b31d1b95c00","Type":"ContainerDied","Data":"501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9"} Jul 16 17:43:27.982296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.982018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86658484bf-sh2n5" event={"ID":"dba5ddab-d31d-4e53-acce-2b31d1b95c00","Type":"ContainerDied","Data":"1c786ffdf9c1fd9a6d4b501ac471e7e04f12166c25fc955e37f80f9e51633993"} Jul 16 17:43:27.982296 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.982039 2564 scope.go:117] "RemoveContainer" containerID="501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9" Jul 16 17:43:27.990894 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.990877 2564 scope.go:117] "RemoveContainer" containerID="501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9" Jul 16 17:43:27.991144 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:43:27.991126 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9\": container with ID starting with 501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9 not found: ID does not exist" containerID="501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9" Jul 16 17:43:27.991201 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:27.991152 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9"} err="failed to get container status \"501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9\": rpc error: code = NotFound desc = could not find container \"501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9\": container with ID starting with 501ae013b0056e2a1bc124c556abb364093ba4311c1a43dfa8d7b100c45b00a9 not found: ID does not exist" Jul 16 17:43:28.004811 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:28.004790 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86658484bf-sh2n5"] Jul 16 17:43:28.010759 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:28.010740 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86658484bf-sh2n5"] Jul 16 17:43:28.574305 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:28.574262 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba5ddab-d31d-4e53-acce-2b31d1b95c00" path="/var/lib/kubelet/pods/dba5ddab-d31d-4e53-acce-2b31d1b95c00/volumes" Jul 16 17:43:32.542980 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.542950 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b7sxm"] Jul 16 17:43:32.543324 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.543271 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba5ddab-d31d-4e53-acce-2b31d1b95c00" containerName="console" Jul 16 17:43:32.543324 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.543281 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba5ddab-d31d-4e53-acce-2b31d1b95c00" containerName="console" Jul 16 17:43:32.543416 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.543351 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba5ddab-d31d-4e53-acce-2b31d1b95c00" containerName="console" Jul 16 17:43:32.547521 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.547505 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.550278 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.550256 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-kc8hl\"" Jul 16 17:43:32.550500 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.550483 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 16 17:43:32.559670 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.559648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b7sxm"] Jul 16 17:43:32.699674 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.699645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhncl\" (UniqueName: \"kubernetes.io/projected/0405a103-1e74-40fc-946b-119e085a9c71-kube-api-access-qhncl\") pod \"odh-model-controller-84b54b7f97-b7sxm\" (UID: \"0405a103-1e74-40fc-946b-119e085a9c71\") " pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.699854 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.699693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0405a103-1e74-40fc-946b-119e085a9c71-cert\") pod \"odh-model-controller-84b54b7f97-b7sxm\" (UID: \"0405a103-1e74-40fc-946b-119e085a9c71\") " pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.800374 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.800289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhncl\" (UniqueName: \"kubernetes.io/projected/0405a103-1e74-40fc-946b-119e085a9c71-kube-api-access-qhncl\") pod \"odh-model-controller-84b54b7f97-b7sxm\" (UID: \"0405a103-1e74-40fc-946b-119e085a9c71\") " pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.800374 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.800333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0405a103-1e74-40fc-946b-119e085a9c71-cert\") pod \"odh-model-controller-84b54b7f97-b7sxm\" (UID: \"0405a103-1e74-40fc-946b-119e085a9c71\") " pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.802658 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.802639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0405a103-1e74-40fc-946b-119e085a9c71-cert\") pod \"odh-model-controller-84b54b7f97-b7sxm\" (UID: \"0405a103-1e74-40fc-946b-119e085a9c71\") " pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.808552 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.808521 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhncl\" (UniqueName: \"kubernetes.io/projected/0405a103-1e74-40fc-946b-119e085a9c71-kube-api-access-qhncl\") pod \"odh-model-controller-84b54b7f97-b7sxm\" (UID: \"0405a103-1e74-40fc-946b-119e085a9c71\") " pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.858011 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.857980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:32.993477 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:32.993451 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b7sxm"] Jul 16 17:43:32.995748 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:43:32.995714 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0405a103_1e74_40fc_946b_119e085a9c71.slice/crio-b80025d976714ce089c485a05e247fdaae8ac86c6ece0991e57fc72f8cb12f70 WatchSource:0}: Error finding container b80025d976714ce089c485a05e247fdaae8ac86c6ece0991e57fc72f8cb12f70: Status 404 returned error can't find the container with id b80025d976714ce089c485a05e247fdaae8ac86c6ece0991e57fc72f8cb12f70 Jul 16 17:43:33.004876 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:33.004851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b7sxm" event={"ID":"0405a103-1e74-40fc-946b-119e085a9c71","Type":"ContainerStarted","Data":"b80025d976714ce089c485a05e247fdaae8ac86c6ece0991e57fc72f8cb12f70"} Jul 16 17:43:36.024068 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:36.024039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b7sxm" event={"ID":"0405a103-1e74-40fc-946b-119e085a9c71","Type":"ContainerStarted","Data":"8ac430400aa1a324de14e41b29fd2372875db9bfdc9bea1d202af03631f4c2a4"} Jul 16 17:43:36.024444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:36.024142 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:43:36.043159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:36.043105 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-b7sxm" podStartSLOduration=1.5203087339999999 podStartE2EDuration="4.043087173s" totalImagesPullingTime="2.522778439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:36.040613868 +0000 UTC m=+594.143717706" watchObservedRunningTime="2026-07-16 17:43:36.043087173 +0000 UTC m=+594.146191013" Jul 16 17:43:42.345656 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:42.345610 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:43:42.346227 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:42.346201 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:43:47.031287 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:43:47.031257 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-b7sxm" Jul 16 17:45:17.085415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.085384 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh"] Jul 16 17:45:17.089009 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.088990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.091559 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.091537 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 16 17:45:17.091559 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.091547 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 17:45:17.091730 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.091602 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Jul 16 17:45:17.091730 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.091648 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wv2xq\"" Jul 16 17:45:17.091959 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.091946 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Jul 16 17:45:17.092040 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.091952 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 16 17:45:17.092509 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.092494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 16 17:45:17.097849 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.097803 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh"] Jul 16 17:45:17.245041 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.245015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdqlc\" (UniqueName: \"kubernetes.io/projected/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-api-access-bdqlc\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.245172 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.245064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.245172 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.245090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.245258 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.245153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.245258 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.245236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.345650 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.345569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.345650 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.345621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdqlc\" (UniqueName: \"kubernetes.io/projected/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-api-access-bdqlc\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.345902 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.345660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.345902 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.345683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.345902 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.345705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.346067 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:45:17.345920 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Jul 16 17:45:17.346067 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:45:17.345985 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls podName:7fcda0fa-aab6-41c4-86b1-d1c566a8477b nodeName:}" failed. No retries permitted until 2026-07-16 17:45:17.845965715 +0000 UTC m=+695.949069536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls") pod "isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" (UID: "7fcda0fa-aab6-41c4-86b1-d1c566a8477b") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Jul 16 17:45:17.346168 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.346095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.346364 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.346344 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.346424 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.346408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.355930 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.355909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdqlc\" (UniqueName: \"kubernetes.io/projected/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-api-access-bdqlc\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.849667 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.849618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:17.851980 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:17.851956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:18.000613 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:18.000578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:18.338396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:18.338341 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh"] Jul 16 17:45:18.341304 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:45:18.341269 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcda0fa_aab6_41c4_86b1_d1c566a8477b.slice/crio-f5bf5ed511d7e2303e93f16642c79858147f56494243af81036ef8486a431da3 WatchSource:0}: Error finding container f5bf5ed511d7e2303e93f16642c79858147f56494243af81036ef8486a431da3: Status 404 returned error can't find the container with id f5bf5ed511d7e2303e93f16642c79858147f56494243af81036ef8486a431da3 Jul 16 17:45:18.491707 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:18.491684 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerStarted","Data":"f5bf5ed511d7e2303e93f16642c79858147f56494243af81036ef8486a431da3"} Jul 16 17:45:22.517580 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:22.517546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerStarted","Data":"1ce0c86e48318bf296042eb3f1dae25f19257523d87a82ac75ffbf885c63aa2a"} Jul 16 17:45:26.536863 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:26.536810 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerID="1ce0c86e48318bf296042eb3f1dae25f19257523d87a82ac75ffbf885c63aa2a" exitCode=0 Jul 16 17:45:26.537312 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:26.536887 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerDied","Data":"1ce0c86e48318bf296042eb3f1dae25f19257523d87a82ac75ffbf885c63aa2a"} Jul 16 17:45:39.399430 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:39.399412 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:45:39.612691 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:39.612639 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerStarted","Data":"20befe9f9ba9b61fdd5b9469b1b9a99e162b8168ba5cb171a05166d0f5ea3024"} Jul 16 17:45:42.629860 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:42.629798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerStarted","Data":"34047efe115e325b394d7d5ad2a769cde6cc9b2977256bcf5afdd76ff78e5dee"} Jul 16 17:45:42.630356 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:42.629931 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:42.652102 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:42.652031 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podStartSLOduration=3.3736894680000002 podStartE2EDuration="25.652018465s" totalImagesPullingTime="18.439595971s" totalInitContainerRuntime="3.838733026s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:42.649999893 +0000 UTC m=+720.753103732" watchObservedRunningTime="2026-07-16 17:45:42.652018465 +0000 UTC m=+720.755122304" Jul 16 17:45:43.634772 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:43.634738 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:43.636538 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:43.636506 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:45:44.639597 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:44.639555 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:45:49.644183 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:49.644152 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:45:49.644889 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:49.644858 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:45:59.645210 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:45:59.645169 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:46:09.644866 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:09.644803 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:46:19.645758 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:19.645707 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:46:29.645381 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:29.645335 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:46:36.494989 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.494950 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl"] Jul 16 17:46:36.497626 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.497603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.500031 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.500004 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-e62f1-kube-rbac-proxy-sar-config\"" Jul 16 17:46:36.500159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.500038 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-e62f1-serving-cert\"" Jul 16 17:46:36.509796 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.509771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl"] Jul 16 17:46:36.528099 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.528077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ce8ffe-1268-4803-b97e-dab63f0a623f-proxy-tls\") pod \"switch-graph-e62f1-6549dd5fcb-8hnkl\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.528204 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.528114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ce8ffe-1268-4803-b97e-dab63f0a623f-openshift-service-ca-bundle\") pod \"switch-graph-e62f1-6549dd5fcb-8hnkl\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.629622 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.629585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ce8ffe-1268-4803-b97e-dab63f0a623f-openshift-service-ca-bundle\") pod \"switch-graph-e62f1-6549dd5fcb-8hnkl\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.629805 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.629753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ce8ffe-1268-4803-b97e-dab63f0a623f-proxy-tls\") pod \"switch-graph-e62f1-6549dd5fcb-8hnkl\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.630238 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.630214 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ce8ffe-1268-4803-b97e-dab63f0a623f-openshift-service-ca-bundle\") pod \"switch-graph-e62f1-6549dd5fcb-8hnkl\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.632170 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.632149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ce8ffe-1268-4803-b97e-dab63f0a623f-proxy-tls\") pod \"switch-graph-e62f1-6549dd5fcb-8hnkl\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.808113 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.808084 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:36.935130 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:36.935072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl"] Jul 16 17:46:36.937511 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:46:36.937481 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ce8ffe_1268_4803_b97e_dab63f0a623f.slice/crio-84b13805a4b43fbb5c021d1bc66ae683dee42a63c7faa5c12e7d651358bc89b6 WatchSource:0}: Error finding container 84b13805a4b43fbb5c021d1bc66ae683dee42a63c7faa5c12e7d651358bc89b6: Status 404 returned error can't find the container with id 84b13805a4b43fbb5c021d1bc66ae683dee42a63c7faa5c12e7d651358bc89b6 Jul 16 17:46:37.880024 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:37.879988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" event={"ID":"83ce8ffe-1268-4803-b97e-dab63f0a623f","Type":"ContainerStarted","Data":"84b13805a4b43fbb5c021d1bc66ae683dee42a63c7faa5c12e7d651358bc89b6"} Jul 16 17:46:39.644810 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:39.644761 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:46:39.890980 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:39.890938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" event={"ID":"83ce8ffe-1268-4803-b97e-dab63f0a623f","Type":"ContainerStarted","Data":"cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e"} Jul 16 17:46:39.891159 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:39.891026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:39.907808 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:39.907693 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podStartSLOduration=1.69264814 podStartE2EDuration="3.907676908s" totalImagesPullingTime="2.215028768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:39.90700394 +0000 UTC m=+778.010107781" watchObservedRunningTime="2026-07-16 17:46:39.907676908 +0000 UTC m=+778.010780790" Jul 16 17:46:45.903330 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:45.903296 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:46:46.590055 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:46.590025 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl"] Jul 16 17:46:46.590210 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:46.590185 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" containerID="cri-o://cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e" gracePeriod=30 Jul 16 17:46:49.645333 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:49.645296 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:46:50.900060 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:50.900019 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:46:55.900651 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:55.900607 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:46:59.646013 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:46:59.645982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:47:00.900340 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:00.900298 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:00.900786 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:00.900405 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:47:05.900363 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:05.900280 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:10.899799 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:10.899758 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:15.899917 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:15.899871 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:16.538800 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.538758 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz"] Jul 16 17:47:16.541899 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.541877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.544149 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.544125 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 16 17:47:16.544272 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.544125 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 16 17:47:16.556613 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.556581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz"] Jul 16 17:47:16.595473 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.595445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b116ced-7b89-49c4-9f18-5d001761774a-openshift-service-ca-bundle\") pod \"model-chainer-c9f7f4f47-wwvjz\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.595577 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.595481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b116ced-7b89-49c4-9f18-5d001761774a-proxy-tls\") pod \"model-chainer-c9f7f4f47-wwvjz\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.696206 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.696174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b116ced-7b89-49c4-9f18-5d001761774a-openshift-service-ca-bundle\") pod \"model-chainer-c9f7f4f47-wwvjz\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.696341 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.696213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b116ced-7b89-49c4-9f18-5d001761774a-proxy-tls\") pod \"model-chainer-c9f7f4f47-wwvjz\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.696799 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.696778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b116ced-7b89-49c4-9f18-5d001761774a-openshift-service-ca-bundle\") pod \"model-chainer-c9f7f4f47-wwvjz\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.698730 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.698712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b116ced-7b89-49c4-9f18-5d001761774a-proxy-tls\") pod \"model-chainer-c9f7f4f47-wwvjz\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.751288 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.751261 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:47:16.796856 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.796779 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ce8ffe-1268-4803-b97e-dab63f0a623f-proxy-tls\") pod \"83ce8ffe-1268-4803-b97e-dab63f0a623f\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " Jul 16 17:47:16.796856 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.796847 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ce8ffe-1268-4803-b97e-dab63f0a623f-openshift-service-ca-bundle\") pod \"83ce8ffe-1268-4803-b97e-dab63f0a623f\" (UID: \"83ce8ffe-1268-4803-b97e-dab63f0a623f\") " Jul 16 17:47:16.797217 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.797190 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ce8ffe-1268-4803-b97e-dab63f0a623f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "83ce8ffe-1268-4803-b97e-dab63f0a623f" (UID: "83ce8ffe-1268-4803-b97e-dab63f0a623f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:47:16.798962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.798944 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ce8ffe-1268-4803-b97e-dab63f0a623f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83ce8ffe-1268-4803-b97e-dab63f0a623f" (UID: "83ce8ffe-1268-4803-b97e-dab63f0a623f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:47:16.852145 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.852118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:16.897610 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.897578 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ce8ffe-1268-4803-b97e-dab63f0a623f-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:16.897610 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.897612 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83ce8ffe-1268-4803-b97e-dab63f0a623f-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:16.979419 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:16.979394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz"] Jul 16 17:47:16.981380 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:47:16.981349 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b116ced_7b89_49c4_9f18_5d001761774a.slice/crio-182bd8bb642373e31b84802f870e62954ce054dcf5ca2c3db8ce39641c09ce0e WatchSource:0}: Error finding container 182bd8bb642373e31b84802f870e62954ce054dcf5ca2c3db8ce39641c09ce0e: Status 404 returned error can't find the container with id 182bd8bb642373e31b84802f870e62954ce054dcf5ca2c3db8ce39641c09ce0e Jul 16 17:47:17.063470 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.063384 2564 generic.go:356] "Generic (PLEG): container finished" podID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerID="cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e" exitCode=0 Jul 16 17:47:17.063470 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.063439 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" Jul 16 17:47:17.063678 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.063470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" event={"ID":"83ce8ffe-1268-4803-b97e-dab63f0a623f","Type":"ContainerDied","Data":"cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e"} Jul 16 17:47:17.063678 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.063525 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl" event={"ID":"83ce8ffe-1268-4803-b97e-dab63f0a623f","Type":"ContainerDied","Data":"84b13805a4b43fbb5c021d1bc66ae683dee42a63c7faa5c12e7d651358bc89b6"} Jul 16 17:47:17.063678 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.063546 2564 scope.go:117] "RemoveContainer" containerID="cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e" Jul 16 17:47:17.065221 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.065199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" event={"ID":"2b116ced-7b89-49c4-9f18-5d001761774a","Type":"ContainerStarted","Data":"6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46"} Jul 16 17:47:17.065331 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.065243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" event={"ID":"2b116ced-7b89-49c4-9f18-5d001761774a","Type":"ContainerStarted","Data":"182bd8bb642373e31b84802f870e62954ce054dcf5ca2c3db8ce39641c09ce0e"} Jul 16 17:47:17.065331 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.065306 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:17.072307 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.072291 2564 scope.go:117] "RemoveContainer" containerID="cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e" Jul 16 17:47:17.072578 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:47:17.072559 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e\": container with ID starting with cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e not found: ID does not exist" containerID="cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e" Jul 16 17:47:17.072642 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.072584 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e"} err="failed to get container status \"cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e\": rpc error: code = NotFound desc = could not find container \"cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e\": container with ID starting with cd8c68c9803a3d3329b7c08cca5b93e03c1f715346f146e1bf28845f84d9a74e not found: ID does not exist" Jul 16 17:47:17.090340 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.090279 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podStartSLOduration=1.090259527 podStartE2EDuration="1.090259527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:47:17.087856664 +0000 UTC m=+815.190960506" watchObservedRunningTime="2026-07-16 17:47:17.090259527 +0000 UTC m=+815.193363373" Jul 16 17:47:17.103401 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.103373 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl"] Jul 16 17:47:17.107939 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:17.107915 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e62f1-6549dd5fcb-8hnkl"] Jul 16 17:47:18.575184 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:18.575148 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" path="/var/lib/kubelet/pods/83ce8ffe-1268-4803-b97e-dab63f0a623f/volumes" Jul 16 17:47:23.078058 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:23.078029 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:26.637707 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:26.637665 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz"] Jul 16 17:47:26.638122 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:26.637902 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" containerID="cri-o://6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46" gracePeriod=30 Jul 16 17:47:27.011247 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:27.011171 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh"] Jul 16 17:47:27.011438 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:27.011395 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" containerID="cri-o://20befe9f9ba9b61fdd5b9469b1b9a99e162b8168ba5cb171a05166d0f5ea3024" gracePeriod=30 Jul 16 17:47:27.011486 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:27.011432 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kube-rbac-proxy" containerID="cri-o://34047efe115e325b394d7d5ad2a769cde6cc9b2977256bcf5afdd76ff78e5dee" gracePeriod=30 Jul 16 17:47:28.076270 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:28.076232 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:28.119817 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:28.119780 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerID="34047efe115e325b394d7d5ad2a769cde6cc9b2977256bcf5afdd76ff78e5dee" exitCode=2 Jul 16 17:47:28.119979 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:28.119859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerDied","Data":"34047efe115e325b394d7d5ad2a769cde6cc9b2977256bcf5afdd76ff78e5dee"} Jul 16 17:47:29.639912 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:29.639870 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Jul 16 17:47:29.644746 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:29.644713 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Jul 16 17:47:31.135948 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.135914 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerID="20befe9f9ba9b61fdd5b9469b1b9a99e162b8168ba5cb171a05166d0f5ea3024" exitCode=0 Jul 16 17:47:31.135948 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.135945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerDied","Data":"20befe9f9ba9b61fdd5b9469b1b9a99e162b8168ba5cb171a05166d0f5ea3024"} Jul 16 17:47:31.164205 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.164175 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:47:31.227974 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.227909 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kserve-provision-location\") pod \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " Jul 16 17:47:31.227974 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.227967 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdqlc\" (UniqueName: \"kubernetes.io/projected/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-api-access-bdqlc\") pod \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " Jul 16 17:47:31.228155 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.228031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls\") pod \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " Jul 16 17:47:31.228155 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.228074 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-cabundle-cert\") pod \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " Jul 16 17:47:31.228155 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.228130 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-rbac-proxy-sar-config\") pod \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\" (UID: \"7fcda0fa-aab6-41c4-86b1-d1c566a8477b\") " Jul 16 17:47:31.228299 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.228236 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fcda0fa-aab6-41c4-86b1-d1c566a8477b" (UID: "7fcda0fa-aab6-41c4-86b1-d1c566a8477b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:47:31.228495 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.228472 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fcda0fa-aab6-41c4-86b1-d1c566a8477b" (UID: "7fcda0fa-aab6-41c4-86b1-d1c566a8477b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:47:31.228579 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.228494 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kserve-provision-location\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:31.228579 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.228546 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7fcda0fa-aab6-41c4-86b1-d1c566a8477b" (UID: "7fcda0fa-aab6-41c4-86b1-d1c566a8477b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:47:31.229989 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.229965 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fcda0fa-aab6-41c4-86b1-d1c566a8477b" (UID: "7fcda0fa-aab6-41c4-86b1-d1c566a8477b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:47:31.230076 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.229966 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-api-access-bdqlc" (OuterVolumeSpecName: "kube-api-access-bdqlc") pod "7fcda0fa-aab6-41c4-86b1-d1c566a8477b" (UID: "7fcda0fa-aab6-41c4-86b1-d1c566a8477b"). InnerVolumeSpecName "kube-api-access-bdqlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:47:31.329349 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.329325 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:31.329349 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.329346 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdqlc\" (UniqueName: \"kubernetes.io/projected/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-kube-api-access-bdqlc\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:31.329484 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.329358 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:31.329484 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:31.329367 2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fcda0fa-aab6-41c4-86b1-d1c566a8477b-cabundle-cert\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:32.141922 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.141891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" event={"ID":"7fcda0fa-aab6-41c4-86b1-d1c566a8477b","Type":"ContainerDied","Data":"f5bf5ed511d7e2303e93f16642c79858147f56494243af81036ef8486a431da3"} Jul 16 17:47:32.142344 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.141932 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh" Jul 16 17:47:32.142344 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.141959 2564 scope.go:117] "RemoveContainer" containerID="34047efe115e325b394d7d5ad2a769cde6cc9b2977256bcf5afdd76ff78e5dee" Jul 16 17:47:32.150813 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.150797 2564 scope.go:117] "RemoveContainer" containerID="20befe9f9ba9b61fdd5b9469b1b9a99e162b8168ba5cb171a05166d0f5ea3024" Jul 16 17:47:32.158480 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.158459 2564 scope.go:117] "RemoveContainer" containerID="1ce0c86e48318bf296042eb3f1dae25f19257523d87a82ac75ffbf885c63aa2a" Jul 16 17:47:32.165765 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.165740 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh"] Jul 16 17:47:32.170257 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.170235 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-c5d76d7f-b9ckh"] Jul 16 17:47:32.574239 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:32.574210 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" path="/var/lib/kubelet/pods/7fcda0fa-aab6-41c4-86b1-d1c566a8477b/volumes" Jul 16 17:47:33.075537 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:33.075498 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:38.076526 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:38.076487 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:38.076903 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:38.076596 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:43.076008 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:43.075966 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:46.836399 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836362 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn"] Jul 16 17:47:46.836802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836729 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" Jul 16 17:47:46.836802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836740 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" Jul 16 17:47:46.836802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836765 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="storage-initializer" Jul 16 17:47:46.836802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836787 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="storage-initializer" Jul 16 17:47:46.836802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836798 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" Jul 16 17:47:46.836802 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836804 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" Jul 16 17:47:46.837010 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836811 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kube-rbac-proxy" Jul 16 17:47:46.837010 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836817 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kube-rbac-proxy" Jul 16 17:47:46.837010 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836893 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kserve-container" Jul 16 17:47:46.837010 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836902 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ce8ffe-1268-4803-b97e-dab63f0a623f" containerName="switch-graph-e62f1" Jul 16 17:47:46.837010 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.836910 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcda0fa-aab6-41c4-86b1-d1c566a8477b" containerName="kube-rbac-proxy" Jul 16 17:47:46.841113 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.841092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:46.843634 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.843614 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0d41-kube-rbac-proxy-sar-config\"" Jul 16 17:47:46.843708 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.843615 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0d41-serving-cert\"" Jul 16 17:47:46.848021 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.847991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn"] Jul 16 17:47:46.957210 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.957171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fe02a67-a134-4110-a5c4-da76d3fcbabe-openshift-service-ca-bundle\") pod \"switch-graph-f0d41-64565558d8-5ldxn\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:46.957382 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:46.957310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe02a67-a134-4110-a5c4-da76d3fcbabe-proxy-tls\") pod \"switch-graph-f0d41-64565558d8-5ldxn\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:47.058682 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:47.058643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe02a67-a134-4110-a5c4-da76d3fcbabe-proxy-tls\") pod \"switch-graph-f0d41-64565558d8-5ldxn\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:47.058851 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:47.058698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fe02a67-a134-4110-a5c4-da76d3fcbabe-openshift-service-ca-bundle\") pod \"switch-graph-f0d41-64565558d8-5ldxn\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:47.059388 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:47.059363 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fe02a67-a134-4110-a5c4-da76d3fcbabe-openshift-service-ca-bundle\") pod \"switch-graph-f0d41-64565558d8-5ldxn\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:47.061291 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:47.061255 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe02a67-a134-4110-a5c4-da76d3fcbabe-proxy-tls\") pod \"switch-graph-f0d41-64565558d8-5ldxn\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:47.152436 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:47.152337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:47.289742 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:47.289712 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn"] Jul 16 17:47:47.292447 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:47:47.292421 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe02a67_a134_4110_a5c4_da76d3fcbabe.slice/crio-c7711ee4764a7ad25448d8246c85a4f9c0b3473ce67889555260d30b72e28404 WatchSource:0}: Error finding container c7711ee4764a7ad25448d8246c85a4f9c0b3473ce67889555260d30b72e28404: Status 404 returned error can't find the container with id c7711ee4764a7ad25448d8246c85a4f9c0b3473ce67889555260d30b72e28404 Jul 16 17:47:48.076319 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:48.076278 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:48.215644 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:48.215607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" event={"ID":"3fe02a67-a134-4110-a5c4-da76d3fcbabe","Type":"ContainerStarted","Data":"feeeb9e4dfc3382b8d7b1d685ce514496b03a4a2520fa0afedac2d8e6c1653bc"} Jul 16 17:47:48.215858 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:48.215665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" event={"ID":"3fe02a67-a134-4110-a5c4-da76d3fcbabe","Type":"ContainerStarted","Data":"c7711ee4764a7ad25448d8246c85a4f9c0b3473ce67889555260d30b72e28404"} Jul 16 17:47:48.215858 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:48.215691 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:48.232698 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:48.232651 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podStartSLOduration=2.232635324 podStartE2EDuration="2.232635324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:47:48.230773261 +0000 UTC m=+846.333877101" watchObservedRunningTime="2026-07-16 17:47:48.232635324 +0000 UTC m=+846.335739163" Jul 16 17:47:53.076468 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:53.076425 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:47:54.226335 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:54.226301 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:47:56.852624 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:56.852596 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:56.946957 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:56.946858 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b116ced-7b89-49c4-9f18-5d001761774a-openshift-service-ca-bundle\") pod \"2b116ced-7b89-49c4-9f18-5d001761774a\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " Jul 16 17:47:56.946957 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:56.946905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b116ced-7b89-49c4-9f18-5d001761774a-proxy-tls\") pod \"2b116ced-7b89-49c4-9f18-5d001761774a\" (UID: \"2b116ced-7b89-49c4-9f18-5d001761774a\") " Jul 16 17:47:56.947529 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:56.947304 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b116ced-7b89-49c4-9f18-5d001761774a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2b116ced-7b89-49c4-9f18-5d001761774a" (UID: "2b116ced-7b89-49c4-9f18-5d001761774a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:47:56.949009 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:56.948984 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b116ced-7b89-49c4-9f18-5d001761774a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b116ced-7b89-49c4-9f18-5d001761774a" (UID: "2b116ced-7b89-49c4-9f18-5d001761774a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:47:57.047584 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.047549 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b116ced-7b89-49c4-9f18-5d001761774a-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:57.047584 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.047579 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b116ced-7b89-49c4-9f18-5d001761774a-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:47:57.259066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.258967 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b116ced-7b89-49c4-9f18-5d001761774a" containerID="6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46" exitCode=137 Jul 16 17:47:57.259066 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.259013 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" Jul 16 17:47:57.259248 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.259059 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" event={"ID":"2b116ced-7b89-49c4-9f18-5d001761774a","Type":"ContainerDied","Data":"6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46"} Jul 16 17:47:57.259248 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.259115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz" event={"ID":"2b116ced-7b89-49c4-9f18-5d001761774a","Type":"ContainerDied","Data":"182bd8bb642373e31b84802f870e62954ce054dcf5ca2c3db8ce39641c09ce0e"} Jul 16 17:47:57.259248 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.259142 2564 scope.go:117] "RemoveContainer" containerID="6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46" Jul 16 17:47:57.268049 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.268026 2564 scope.go:117] "RemoveContainer" containerID="6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46" Jul 16 17:47:57.268351 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:47:57.268332 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46\": container with ID starting with 6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46 not found: ID does not exist" containerID="6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46" Jul 16 17:47:57.268415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.268360 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46"} err="failed to get container status \"6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46\": rpc error: code = NotFound desc = could not find container \"6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46\": container with ID starting with 6887ebed8371f5db95562abed89fc30fc587732a8158a0189b686a1d92163f46 not found: ID does not exist" Jul 16 17:47:57.280636 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.280604 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz"] Jul 16 17:47:57.284502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:57.284478 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c9f7f4f47-wwvjz"] Jul 16 17:47:58.574610 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:47:58.574577 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" path="/var/lib/kubelet/pods/2b116ced-7b89-49c4-9f18-5d001761774a/volumes" Jul 16 17:48:26.854906 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.854876 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64"] Jul 16 17:48:26.855345 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.855265 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" Jul 16 17:48:26.855345 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.855275 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" Jul 16 17:48:26.855345 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.855343 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b116ced-7b89-49c4-9f18-5d001761774a" containerName="model-chainer" Jul 16 17:48:26.858439 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.858423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:26.860721 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.860692 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-38e88-kube-rbac-proxy-sar-config\"" Jul 16 17:48:26.860865 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.860688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-38e88-serving-cert\"" Jul 16 17:48:26.866718 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.866693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64"] Jul 16 17:48:26.975957 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.975915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403e454a-70f5-4888-8482-66cc1742652e-openshift-service-ca-bundle\") pod \"sequence-graph-38e88-5fb9579648-khh64\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:26.976154 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:26.975994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e454a-70f5-4888-8482-66cc1742652e-proxy-tls\") pod \"sequence-graph-38e88-5fb9579648-khh64\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:27.077464 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.077424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403e454a-70f5-4888-8482-66cc1742652e-openshift-service-ca-bundle\") pod \"sequence-graph-38e88-5fb9579648-khh64\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:27.077681 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.077487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e454a-70f5-4888-8482-66cc1742652e-proxy-tls\") pod \"sequence-graph-38e88-5fb9579648-khh64\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:27.078205 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.078177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403e454a-70f5-4888-8482-66cc1742652e-openshift-service-ca-bundle\") pod \"sequence-graph-38e88-5fb9579648-khh64\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:27.079957 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.079938 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e454a-70f5-4888-8482-66cc1742652e-proxy-tls\") pod \"sequence-graph-38e88-5fb9579648-khh64\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:27.169333 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.169247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:27.302936 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.302910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64"] Jul 16 17:48:27.305394 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:48:27.305369 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403e454a_70f5_4888_8482_66cc1742652e.slice/crio-145bddfaf773fe6d63093bfb71e3d4f23f9c558202ae415963f24952948e5c54 WatchSource:0}: Error finding container 145bddfaf773fe6d63093bfb71e3d4f23f9c558202ae415963f24952948e5c54: Status 404 returned error can't find the container with id 145bddfaf773fe6d63093bfb71e3d4f23f9c558202ae415963f24952948e5c54 Jul 16 17:48:27.398169 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.398126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" event={"ID":"403e454a-70f5-4888-8482-66cc1742652e","Type":"ContainerStarted","Data":"41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30"} Jul 16 17:48:27.398343 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.398193 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" event={"ID":"403e454a-70f5-4888-8482-66cc1742652e","Type":"ContainerStarted","Data":"145bddfaf773fe6d63093bfb71e3d4f23f9c558202ae415963f24952948e5c54"} Jul 16 17:48:27.398343 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.398235 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:27.421489 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:27.421394 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podStartSLOduration=1.421378968 podStartE2EDuration="1.421378968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:27.420886938 +0000 UTC m=+885.523990778" watchObservedRunningTime="2026-07-16 17:48:27.421378968 +0000 UTC m=+885.524482807" Jul 16 17:48:33.409352 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:33.409322 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:48:42.373942 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:42.373866 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:48:42.375426 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:48:42.375402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:53:42.407900 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:53:42.407807 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:53:42.411045 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:53:42.411024 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:56:01.600334 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:01.600298 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn"] Jul 16 17:56:01.602883 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:01.600546 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" containerID="cri-o://feeeb9e4dfc3382b8d7b1d685ce514496b03a4a2520fa0afedac2d8e6c1653bc" gracePeriod=30 Jul 16 17:56:04.224745 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:04.224655 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:09.225132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:09.225084 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:14.225344 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:14.225298 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:14.225788 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:14.225418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:56:19.224976 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:19.224936 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:24.225397 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:24.225347 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:29.225701 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:29.225633 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:31.626753 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:31.626703 2564 generic.go:356] "Generic (PLEG): container finished" podID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerID="feeeb9e4dfc3382b8d7b1d685ce514496b03a4a2520fa0afedac2d8e6c1653bc" exitCode=0 Jul 16 17:56:31.627086 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:31.626776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" event={"ID":"3fe02a67-a134-4110-a5c4-da76d3fcbabe","Type":"ContainerDied","Data":"feeeb9e4dfc3382b8d7b1d685ce514496b03a4a2520fa0afedac2d8e6c1653bc"} Jul 16 17:56:31.764722 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:31.764701 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:56:31.924108 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:31.924044 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe02a67-a134-4110-a5c4-da76d3fcbabe-proxy-tls\") pod \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " Jul 16 17:56:31.924235 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:31.924142 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fe02a67-a134-4110-a5c4-da76d3fcbabe-openshift-service-ca-bundle\") pod \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\" (UID: \"3fe02a67-a134-4110-a5c4-da76d3fcbabe\") " Jul 16 17:56:31.924472 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:31.924442 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe02a67-a134-4110-a5c4-da76d3fcbabe-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3fe02a67-a134-4110-a5c4-da76d3fcbabe" (UID: "3fe02a67-a134-4110-a5c4-da76d3fcbabe"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:56:31.926207 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:31.926186 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe02a67-a134-4110-a5c4-da76d3fcbabe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fe02a67-a134-4110-a5c4-da76d3fcbabe" (UID: "3fe02a67-a134-4110-a5c4-da76d3fcbabe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:56:32.024715 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:32.024688 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fe02a67-a134-4110-a5c4-da76d3fcbabe-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:56:32.024715 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:32.024712 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fe02a67-a134-4110-a5c4-da76d3fcbabe-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:56:32.632962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:32.632935 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" Jul 16 17:56:32.632962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:32.632950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn" event={"ID":"3fe02a67-a134-4110-a5c4-da76d3fcbabe","Type":"ContainerDied","Data":"c7711ee4764a7ad25448d8246c85a4f9c0b3473ce67889555260d30b72e28404"} Jul 16 17:56:32.633440 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:32.633007 2564 scope.go:117] "RemoveContainer" containerID="feeeb9e4dfc3382b8d7b1d685ce514496b03a4a2520fa0afedac2d8e6c1653bc" Jul 16 17:56:32.651704 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:32.651680 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn"] Jul 16 17:56:32.655695 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:32.655671 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d41-64565558d8-5ldxn"] Jul 16 17:56:34.575728 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:34.575688 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" path="/var/lib/kubelet/pods/3fe02a67-a134-4110-a5c4-da76d3fcbabe/volumes" Jul 16 17:56:41.468521 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:41.468479 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64"] Jul 16 17:56:41.469031 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:41.468683 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" containerID="cri-o://41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30" gracePeriod=30 Jul 16 17:56:43.407297 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:43.407248 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:48.406646 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:48.406601 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:53.407541 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:53.407495 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:56:53.408069 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:53.407621 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:56:58.407727 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:56:58.407682 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:01.875847 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.875794 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc"] Jul 16 17:57:01.876365 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.876343 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" Jul 16 17:57:01.876449 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.876369 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" Jul 16 17:57:01.876502 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.876459 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe02a67-a134-4110-a5c4-da76d3fcbabe" containerName="switch-graph-f0d41" Jul 16 17:57:01.880763 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.880739 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:01.882946 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.882926 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1da67-kube-rbac-proxy-sar-config\"" Jul 16 17:57:01.883032 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.882959 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1da67-serving-cert\"" Jul 16 17:57:01.895673 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.895647 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc"] Jul 16 17:57:01.957453 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.957413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5706312-1686-4264-b434-6d87298148b3-openshift-service-ca-bundle\") pod \"ensemble-graph-1da67-9557f86c-ptfbc\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:01.957604 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:01.957517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5706312-1686-4264-b434-6d87298148b3-proxy-tls\") pod \"ensemble-graph-1da67-9557f86c-ptfbc\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:02.058239 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.058209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5706312-1686-4264-b434-6d87298148b3-proxy-tls\") pod \"ensemble-graph-1da67-9557f86c-ptfbc\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:02.058396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.058293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5706312-1686-4264-b434-6d87298148b3-openshift-service-ca-bundle\") pod \"ensemble-graph-1da67-9557f86c-ptfbc\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:02.058980 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.058954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5706312-1686-4264-b434-6d87298148b3-openshift-service-ca-bundle\") pod \"ensemble-graph-1da67-9557f86c-ptfbc\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:02.060472 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.060450 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5706312-1686-4264-b434-6d87298148b3-proxy-tls\") pod \"ensemble-graph-1da67-9557f86c-ptfbc\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:02.191438 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.191371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:02.324632 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.324607 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc"] Jul 16 17:57:02.327049 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:57:02.327020 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5706312_1686_4264_b434_6d87298148b3.slice/crio-8aebb8012026b20d4275012c6420179cbc5064546222c274214de177d197d331 WatchSource:0}: Error finding container 8aebb8012026b20d4275012c6420179cbc5064546222c274214de177d197d331: Status 404 returned error can't find the container with id 8aebb8012026b20d4275012c6420179cbc5064546222c274214de177d197d331 Jul 16 17:57:02.772819 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.772780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" event={"ID":"c5706312-1686-4264-b434-6d87298148b3","Type":"ContainerStarted","Data":"78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66"} Jul 16 17:57:02.773030 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.772862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" event={"ID":"c5706312-1686-4264-b434-6d87298148b3","Type":"ContainerStarted","Data":"8aebb8012026b20d4275012c6420179cbc5064546222c274214de177d197d331"} Jul 16 17:57:02.773030 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.772977 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:02.789731 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:02.789686 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podStartSLOduration=1.789671816 podStartE2EDuration="1.789671816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:02.789210335 +0000 UTC m=+1400.892314178" watchObservedRunningTime="2026-07-16 17:57:02.789671816 +0000 UTC m=+1400.892775634" Jul 16 17:57:03.407613 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:03.407563 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:08.406687 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:08.406647 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:08.784072 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:08.784046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:11.626650 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.626626 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:57:11.735062 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.734997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e454a-70f5-4888-8482-66cc1742652e-proxy-tls\") pod \"403e454a-70f5-4888-8482-66cc1742652e\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " Jul 16 17:57:11.735192 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.735068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403e454a-70f5-4888-8482-66cc1742652e-openshift-service-ca-bundle\") pod \"403e454a-70f5-4888-8482-66cc1742652e\" (UID: \"403e454a-70f5-4888-8482-66cc1742652e\") " Jul 16 17:57:11.735425 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.735404 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403e454a-70f5-4888-8482-66cc1742652e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "403e454a-70f5-4888-8482-66cc1742652e" (UID: "403e454a-70f5-4888-8482-66cc1742652e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:57:11.737081 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.737049 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403e454a-70f5-4888-8482-66cc1742652e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "403e454a-70f5-4888-8482-66cc1742652e" (UID: "403e454a-70f5-4888-8482-66cc1742652e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:11.812952 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.812925 2564 generic.go:356] "Generic (PLEG): container finished" podID="403e454a-70f5-4888-8482-66cc1742652e" containerID="41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30" exitCode=0 Jul 16 17:57:11.813059 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.812974 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" Jul 16 17:57:11.813059 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.813008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" event={"ID":"403e454a-70f5-4888-8482-66cc1742652e","Type":"ContainerDied","Data":"41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30"} Jul 16 17:57:11.813132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.813065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64" event={"ID":"403e454a-70f5-4888-8482-66cc1742652e","Type":"ContainerDied","Data":"145bddfaf773fe6d63093bfb71e3d4f23f9c558202ae415963f24952948e5c54"} Jul 16 17:57:11.813132 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.813094 2564 scope.go:117] "RemoveContainer" containerID="41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30" Jul 16 17:57:11.821479 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.821464 2564 scope.go:117] "RemoveContainer" containerID="41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30" Jul 16 17:57:11.821736 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:57:11.821718 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30\": container with ID starting with 41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30 not found: ID does not exist" containerID="41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30" Jul 16 17:57:11.821788 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.821743 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30"} err="failed to get container status \"41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30\": rpc error: code = NotFound desc = could not find container \"41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30\": container with ID starting with 41c05f5f5978ce8dcf0237d5b87d6d6d06e2ee8cda5854649c328fd9590e2c30 not found: ID does not exist" Jul 16 17:57:11.835762 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.835738 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403e454a-70f5-4888-8482-66cc1742652e-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:57:11.835891 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.835766 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e454a-70f5-4888-8482-66cc1742652e-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:57:11.837604 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.837580 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64"] Jul 16 17:57:11.850084 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:11.850064 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-38e88-5fb9579648-khh64"] Jul 16 17:57:12.055250 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:12.055217 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc"] Jul 16 17:57:12.055454 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:12.055408 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" containerID="cri-o://78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66" gracePeriod=30 Jul 16 17:57:12.575434 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:12.575402 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403e454a-70f5-4888-8482-66cc1742652e" path="/var/lib/kubelet/pods/403e454a-70f5-4888-8482-66cc1742652e/volumes" Jul 16 17:57:13.782468 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:13.782434 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:18.781730 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:18.781685 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:23.782371 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:23.782330 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:23.782883 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:23.782440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:28.782354 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:28.782309 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:33.782152 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:33.782117 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:38.781942 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:38.781898 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:41.683437 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.683396 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt"] Jul 16 17:57:41.683873 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.683783 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" Jul 16 17:57:41.683873 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.683795 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" Jul 16 17:57:41.683948 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.683891 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="403e454a-70f5-4888-8482-66cc1742652e" containerName="sequence-graph-38e88" Jul 16 17:57:41.687176 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.687156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:41.689461 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.689430 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-6687f-serving-cert\"" Jul 16 17:57:41.689461 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.689450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-6687f-kube-rbac-proxy-sar-config\"" Jul 16 17:57:41.697082 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.697055 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt"] Jul 16 17:57:41.785617 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.785573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-openshift-service-ca-bundle\") pod \"sequence-graph-6687f-85bb45f787-r8tvt\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:41.785806 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.785629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls\") pod \"sequence-graph-6687f-85bb45f787-r8tvt\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:41.886116 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.886077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-openshift-service-ca-bundle\") pod \"sequence-graph-6687f-85bb45f787-r8tvt\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:41.886311 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.886140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls\") pod \"sequence-graph-6687f-85bb45f787-r8tvt\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:41.886311 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:57:41.886239 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-6687f-serving-cert: secret "sequence-graph-6687f-serving-cert" not found Jul 16 17:57:41.886311 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:57:41.886310 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls podName:9f1b324f-0947-4543-8f2c-d0bb33ae79c1 nodeName:}" failed. No retries permitted until 2026-07-16 17:57:42.386293637 +0000 UTC m=+1440.489397471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls") pod "sequence-graph-6687f-85bb45f787-r8tvt" (UID: "9f1b324f-0947-4543-8f2c-d0bb33ae79c1") : secret "sequence-graph-6687f-serving-cert" not found Jul 16 17:57:41.886795 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:41.886769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-openshift-service-ca-bundle\") pod \"sequence-graph-6687f-85bb45f787-r8tvt\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:42.213530 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.213508 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:42.288921 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.288888 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5706312-1686-4264-b434-6d87298148b3-openshift-service-ca-bundle\") pod \"c5706312-1686-4264-b434-6d87298148b3\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " Jul 16 17:57:42.289118 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.288947 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5706312-1686-4264-b434-6d87298148b3-proxy-tls\") pod \"c5706312-1686-4264-b434-6d87298148b3\" (UID: \"c5706312-1686-4264-b434-6d87298148b3\") " Jul 16 17:57:42.289259 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.289235 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5706312-1686-4264-b434-6d87298148b3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c5706312-1686-4264-b434-6d87298148b3" (UID: "c5706312-1686-4264-b434-6d87298148b3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:57:42.290962 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.290937 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5706312-1686-4264-b434-6d87298148b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5706312-1686-4264-b434-6d87298148b3" (UID: "c5706312-1686-4264-b434-6d87298148b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:42.390396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.390301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls\") pod \"sequence-graph-6687f-85bb45f787-r8tvt\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:42.390396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.390363 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5706312-1686-4264-b434-6d87298148b3-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:57:42.390570 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.390448 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5706312-1686-4264-b434-6d87298148b3-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:57:42.392648 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.392621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls\") pod \"sequence-graph-6687f-85bb45f787-r8tvt\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:42.598354 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.598332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:42.727445 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.727420 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt"] Jul 16 17:57:42.729743 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:57:42.729717 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1b324f_0947_4543_8f2c_d0bb33ae79c1.slice/crio-d96790c851db8e8e16d2dc270a40bcee809aa4cbc9aeaad81dd542e091c7911c WatchSource:0}: Error finding container d96790c851db8e8e16d2dc270a40bcee809aa4cbc9aeaad81dd542e091c7911c: Status 404 returned error can't find the container with id d96790c851db8e8e16d2dc270a40bcee809aa4cbc9aeaad81dd542e091c7911c Jul 16 17:57:42.966860 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.966753 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5706312-1686-4264-b434-6d87298148b3" containerID="78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66" exitCode=0 Jul 16 17:57:42.966860 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.966796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" event={"ID":"c5706312-1686-4264-b434-6d87298148b3","Type":"ContainerDied","Data":"78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66"} Jul 16 17:57:42.967071 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.966869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" event={"ID":"c5706312-1686-4264-b434-6d87298148b3","Type":"ContainerDied","Data":"8aebb8012026b20d4275012c6420179cbc5064546222c274214de177d197d331"} Jul 16 17:57:42.967071 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.966804 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc" Jul 16 17:57:42.967071 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.966904 2564 scope.go:117] "RemoveContainer" containerID="78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66" Jul 16 17:57:42.968645 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.968614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" event={"ID":"9f1b324f-0947-4543-8f2c-d0bb33ae79c1","Type":"ContainerStarted","Data":"e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd"} Jul 16 17:57:42.968790 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.968669 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" event={"ID":"9f1b324f-0947-4543-8f2c-d0bb33ae79c1","Type":"ContainerStarted","Data":"d96790c851db8e8e16d2dc270a40bcee809aa4cbc9aeaad81dd542e091c7911c"} Jul 16 17:57:42.968790 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.968700 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:42.975649 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.975631 2564 scope.go:117] "RemoveContainer" containerID="78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66" Jul 16 17:57:42.975899 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:57:42.975879 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66\": container with ID starting with 78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66 not found: ID does not exist" containerID="78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66" Jul 16 17:57:42.975991 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.975906 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66"} err="failed to get container status \"78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66\": rpc error: code = NotFound desc = could not find container \"78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66\": container with ID starting with 78f3b427b5b8814882c8003996dd0815fa55ec31c036fa95cbe6272c7e790b66 not found: ID does not exist" Jul 16 17:57:42.984386 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.984366 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc"] Jul 16 17:57:42.988707 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:42.988687 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1da67-9557f86c-ptfbc"] Jul 16 17:57:43.005048 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:43.005012 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podStartSLOduration=2.004999844 podStartE2EDuration="2.004999844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:43.00372106 +0000 UTC m=+1441.106824897" watchObservedRunningTime="2026-07-16 17:57:43.004999844 +0000 UTC m=+1441.108103683" Jul 16 17:57:44.575396 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:44.575366 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5706312-1686-4264-b434-6d87298148b3" path="/var/lib/kubelet/pods/c5706312-1686-4264-b434-6d87298148b3/volumes" Jul 16 17:57:48.982409 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:48.982366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:57:51.777012 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:51.776983 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt"] Jul 16 17:57:51.777431 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:51.777156 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" containerID="cri-o://e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd" gracePeriod=30 Jul 16 17:57:53.980444 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:53.980394 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:57:58.980320 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:57:58.980281 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:58:03.980101 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:03.980057 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:58:03.980519 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:03.980190 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:58:08.980096 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:08.980049 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:58:12.342003 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.341969 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n"] Jul 16 17:58:12.342409 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.342395 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" Jul 16 17:58:12.342454 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.342412 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" Jul 16 17:58:12.342496 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.342484 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5706312-1686-4264-b434-6d87298148b3" containerName="ensemble-graph-1da67" Jul 16 17:58:12.345213 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.345197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:12.348070 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.348046 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-efe2d-kube-rbac-proxy-sar-config\"" Jul 16 17:58:12.348070 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.348043 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-efe2d-serving-cert\"" Jul 16 17:58:12.360527 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.360501 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n"] Jul 16 17:58:12.448003 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.447971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-openshift-service-ca-bundle\") pod \"ensemble-graph-efe2d-c6c68b6dc-xnx4n\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:12.448127 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.448053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls\") pod \"ensemble-graph-efe2d-c6c68b6dc-xnx4n\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:12.549308 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.549276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-openshift-service-ca-bundle\") pod \"ensemble-graph-efe2d-c6c68b6dc-xnx4n\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:12.549459 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.549338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls\") pod \"ensemble-graph-efe2d-c6c68b6dc-xnx4n\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:12.549459 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:58:12.549453 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-efe2d-serving-cert: secret "ensemble-graph-efe2d-serving-cert" not found Jul 16 17:58:12.549590 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:58:12.549517 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls podName:61f7e3d7-08fa-422a-ba60-5dbf75eff86c nodeName:}" failed. No retries permitted until 2026-07-16 17:58:13.049499435 +0000 UTC m=+1471.152603252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls") pod "ensemble-graph-efe2d-c6c68b6dc-xnx4n" (UID: "61f7e3d7-08fa-422a-ba60-5dbf75eff86c") : secret "ensemble-graph-efe2d-serving-cert" not found Jul 16 17:58:12.550048 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:12.550023 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-openshift-service-ca-bundle\") pod \"ensemble-graph-efe2d-c6c68b6dc-xnx4n\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:13.054486 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:13.054439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls\") pod \"ensemble-graph-efe2d-c6c68b6dc-xnx4n\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:13.056777 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:13.056750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls\") pod \"ensemble-graph-efe2d-c6c68b6dc-xnx4n\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:13.256000 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:13.255967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:13.383764 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:13.383742 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n"] Jul 16 17:58:13.386091 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:58:13.386057 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f7e3d7_08fa_422a_ba60_5dbf75eff86c.slice/crio-1fcc15e92a730ccdf323062df9962c1a2559e6a2cb530cd889d6d41b4943df95 WatchSource:0}: Error finding container 1fcc15e92a730ccdf323062df9962c1a2559e6a2cb530cd889d6d41b4943df95: Status 404 returned error can't find the container with id 1fcc15e92a730ccdf323062df9962c1a2559e6a2cb530cd889d6d41b4943df95 Jul 16 17:58:13.980107 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:13.980070 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:58:14.113643 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:14.113593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" event={"ID":"61f7e3d7-08fa-422a-ba60-5dbf75eff86c","Type":"ContainerStarted","Data":"1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173"} Jul 16 17:58:14.113784 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:14.113660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" event={"ID":"61f7e3d7-08fa-422a-ba60-5dbf75eff86c","Type":"ContainerStarted","Data":"1fcc15e92a730ccdf323062df9962c1a2559e6a2cb530cd889d6d41b4943df95"} Jul 16 17:58:14.113784 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:14.113689 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:14.130679 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:14.130633 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podStartSLOduration=2.130619775 podStartE2EDuration="2.130619775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:58:14.130369805 +0000 UTC m=+1472.233473644" watchObservedRunningTime="2026-07-16 17:58:14.130619775 +0000 UTC m=+1472.233723615" Jul 16 17:58:18.980134 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:18.980082 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:58:20.124334 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:20.124296 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 17:58:21.924575 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:21.924552 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:58:22.031619 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.031557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls\") pod \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " Jul 16 17:58:22.031748 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.031647 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-openshift-service-ca-bundle\") pod \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\" (UID: \"9f1b324f-0947-4543-8f2c-d0bb33ae79c1\") " Jul 16 17:58:22.031995 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.031970 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9f1b324f-0947-4543-8f2c-d0bb33ae79c1" (UID: "9f1b324f-0947-4543-8f2c-d0bb33ae79c1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:58:22.033443 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.033423 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f1b324f-0947-4543-8f2c-d0bb33ae79c1" (UID: "9f1b324f-0947-4543-8f2c-d0bb33ae79c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:58:22.132998 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.132975 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:58:22.132998 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.132996 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f1b324f-0947-4543-8f2c-d0bb33ae79c1-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 17:58:22.157514 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.157485 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerID="e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd" exitCode=0 Jul 16 17:58:22.157598 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.157517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" event={"ID":"9f1b324f-0947-4543-8f2c-d0bb33ae79c1","Type":"ContainerDied","Data":"e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd"} Jul 16 17:58:22.157598 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.157528 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" Jul 16 17:58:22.157598 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.157551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt" event={"ID":"9f1b324f-0947-4543-8f2c-d0bb33ae79c1","Type":"ContainerDied","Data":"d96790c851db8e8e16d2dc270a40bcee809aa4cbc9aeaad81dd542e091c7911c"} Jul 16 17:58:22.157598 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.157572 2564 scope.go:117] "RemoveContainer" containerID="e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd" Jul 16 17:58:22.166040 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.166016 2564 scope.go:117] "RemoveContainer" containerID="e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd" Jul 16 17:58:22.166278 ip-10-0-135-249 kubenswrapper[2564]: E0716 17:58:22.166259 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd\": container with ID starting with e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd not found: ID does not exist" containerID="e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd" Jul 16 17:58:22.166325 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.166287 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd"} err="failed to get container status \"e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd\": rpc error: code = NotFound desc = could not find container \"e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd\": container with ID starting with e587ffea51457e08850d39bc2c846272f6168cb598573e0346541c3d5d19bfcd not found: ID does not exist" Jul 16 17:58:22.192585 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.192566 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt"] Jul 16 17:58:22.197891 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.197870 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6687f-85bb45f787-r8tvt"] Jul 16 17:58:22.577091 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:22.577059 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" path="/var/lib/kubelet/pods/9f1b324f-0947-4543-8f2c-d0bb33ae79c1/volumes" Jul 16 17:58:42.436711 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:42.436679 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:58:42.441372 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:42.441354 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 17:58:52.008605 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.008555 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6"] Jul 16 17:58:52.009239 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.009215 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" Jul 16 17:58:52.009324 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.009241 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" Jul 16 17:58:52.009379 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.009366 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1b324f-0947-4543-8f2c-d0bb33ae79c1" containerName="sequence-graph-6687f" Jul 16 17:58:52.012559 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.012539 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.014637 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.014616 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a90bb-serving-cert\"" Jul 16 17:58:52.014637 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.014625 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a90bb-kube-rbac-proxy-sar-config\"" Jul 16 17:58:52.020868 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.020848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6"] Jul 16 17:58:52.054924 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.054899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1a1886-b7e2-4b8e-9812-feb2615632a8-openshift-service-ca-bundle\") pod \"sequence-graph-a90bb-5b767cc4d9-qrwl6\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.055015 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.054944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1a1886-b7e2-4b8e-9812-feb2615632a8-proxy-tls\") pod \"sequence-graph-a90bb-5b767cc4d9-qrwl6\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.156426 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.156390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1a1886-b7e2-4b8e-9812-feb2615632a8-proxy-tls\") pod \"sequence-graph-a90bb-5b767cc4d9-qrwl6\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.156575 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.156468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1a1886-b7e2-4b8e-9812-feb2615632a8-openshift-service-ca-bundle\") pod \"sequence-graph-a90bb-5b767cc4d9-qrwl6\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.157048 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.157026 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1a1886-b7e2-4b8e-9812-feb2615632a8-openshift-service-ca-bundle\") pod \"sequence-graph-a90bb-5b767cc4d9-qrwl6\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.158779 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.158757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1a1886-b7e2-4b8e-9812-feb2615632a8-proxy-tls\") pod \"sequence-graph-a90bb-5b767cc4d9-qrwl6\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.323422 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.323395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:52.453415 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:52.453397 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6"] Jul 16 17:58:52.455660 ip-10-0-135-249 kubenswrapper[2564]: W0716 17:58:52.455628 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1a1886_b7e2_4b8e_9812_feb2615632a8.slice/crio-6efabfd4eba22fc53167f5bef04a12a63e4c9634038c0aa08dd38899d4f261d6 WatchSource:0}: Error finding container 6efabfd4eba22fc53167f5bef04a12a63e4c9634038c0aa08dd38899d4f261d6: Status 404 returned error can't find the container with id 6efabfd4eba22fc53167f5bef04a12a63e4c9634038c0aa08dd38899d4f261d6 Jul 16 17:58:53.304115 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:53.304080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" event={"ID":"6a1a1886-b7e2-4b8e-9812-feb2615632a8","Type":"ContainerStarted","Data":"14f1c4fe4ad4035335d51b9956410f771bbdbf867e0e300e7ba13ec7cb405b52"} Jul 16 17:58:53.304528 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:53.304133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" event={"ID":"6a1a1886-b7e2-4b8e-9812-feb2615632a8","Type":"ContainerStarted","Data":"6efabfd4eba22fc53167f5bef04a12a63e4c9634038c0aa08dd38899d4f261d6"} Jul 16 17:58:53.304528 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:53.304162 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 17:58:53.320513 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:53.320467 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podStartSLOduration=2.32045551 podStartE2EDuration="2.32045551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:58:53.318416086 +0000 UTC m=+1511.421519927" watchObservedRunningTime="2026-07-16 17:58:53.32045551 +0000 UTC m=+1511.423559349" Jul 16 17:58:59.315046 ip-10-0-135-249 kubenswrapper[2564]: I0716 17:58:59.315012 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 18:03:42.465188 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:03:42.465158 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:03:42.469665 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:03:42.469645 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:06:26.975620 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:26.975579 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n"] Jul 16 18:06:26.976168 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:26.975863 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" containerID="cri-o://1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173" gracePeriod=30 Jul 16 18:06:30.122078 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:30.122038 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:06:35.122613 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:35.122513 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:06:40.122134 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:40.122092 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:06:40.122543 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:40.122200 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 18:06:45.122409 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:45.122356 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:06:50.122778 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:50.122732 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:06:55.122650 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:55.122602 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:06:57.146154 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.146126 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 18:06:57.200780 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.200731 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls\") pod \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " Jul 16 18:06:57.200780 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.200787 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-openshift-service-ca-bundle\") pod \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\" (UID: \"61f7e3d7-08fa-422a-ba60-5dbf75eff86c\") " Jul 16 18:06:57.201186 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.201159 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "61f7e3d7-08fa-422a-ba60-5dbf75eff86c" (UID: "61f7e3d7-08fa-422a-ba60-5dbf75eff86c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:06:57.203104 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.203073 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61f7e3d7-08fa-422a-ba60-5dbf75eff86c" (UID: "61f7e3d7-08fa-422a-ba60-5dbf75eff86c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:06:57.301896 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.301816 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:06:57.301896 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.301895 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61f7e3d7-08fa-422a-ba60-5dbf75eff86c-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:06:57.536883 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.536817 2564 generic.go:356] "Generic (PLEG): container finished" podID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerID="1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173" exitCode=0 Jul 16 18:06:57.536883 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.536871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" event={"ID":"61f7e3d7-08fa-422a-ba60-5dbf75eff86c","Type":"ContainerDied","Data":"1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173"} Jul 16 18:06:57.537127 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.536929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" event={"ID":"61f7e3d7-08fa-422a-ba60-5dbf75eff86c","Type":"ContainerDied","Data":"1fcc15e92a730ccdf323062df9962c1a2559e6a2cb530cd889d6d41b4943df95"} Jul 16 18:06:57.537127 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.536886 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n" Jul 16 18:06:57.537127 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.536953 2564 scope.go:117] "RemoveContainer" containerID="1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173" Jul 16 18:06:57.546268 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.546247 2564 scope.go:117] "RemoveContainer" containerID="1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173" Jul 16 18:06:57.546615 ip-10-0-135-249 kubenswrapper[2564]: E0716 18:06:57.546588 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173\": container with ID starting with 1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173 not found: ID does not exist" containerID="1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173" Jul 16 18:06:57.546705 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.546625 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173"} err="failed to get container status \"1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173\": rpc error: code = NotFound desc = could not find container \"1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173\": container with ID starting with 1b3a179b96abd06936da6541126fe34d5460121c6440d0cad4428ef5d950a173 not found: ID does not exist" Jul 16 18:06:57.560985 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.560890 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n"] Jul 16 18:06:57.564575 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:57.564547 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-efe2d-c6c68b6dc-xnx4n"] Jul 16 18:06:58.580685 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:06:58.575345 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" path="/var/lib/kubelet/pods/61f7e3d7-08fa-422a-ba60-5dbf75eff86c/volumes" Jul 16 18:07:06.565665 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:06.565624 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6"] Jul 16 18:07:06.566126 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:06.565882 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" containerID="cri-o://14f1c4fe4ad4035335d51b9956410f771bbdbf867e0e300e7ba13ec7cb405b52" gracePeriod=30 Jul 16 18:07:09.313427 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:09.313382 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:14.313684 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:14.313636 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:19.313860 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:19.313783 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:19.314277 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:19.313936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 18:07:24.313001 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:24.312956 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:27.240904 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.240870 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b"] Jul 16 18:07:27.241272 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.241236 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" Jul 16 18:07:27.241272 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.241247 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" Jul 16 18:07:27.241344 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.241310 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f7e3d7-08fa-422a-ba60-5dbf75eff86c" containerName="ensemble-graph-efe2d" Jul 16 18:07:27.244230 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.244209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.246922 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.246889 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a019b-serving-cert\"" Jul 16 18:07:27.246922 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.246889 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a019b-kube-rbac-proxy-sar-config\"" Jul 16 18:07:27.256537 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.256515 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b"] Jul 16 18:07:27.377842 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.377801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072639f0-8e17-4af7-925d-c3d62da94102-proxy-tls\") pod \"splitter-graph-a019b-69676cb5c4-5td7b\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.378004 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.377865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072639f0-8e17-4af7-925d-c3d62da94102-openshift-service-ca-bundle\") pod \"splitter-graph-a019b-69676cb5c4-5td7b\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.478588 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.478555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072639f0-8e17-4af7-925d-c3d62da94102-proxy-tls\") pod \"splitter-graph-a019b-69676cb5c4-5td7b\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.478786 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.478598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072639f0-8e17-4af7-925d-c3d62da94102-openshift-service-ca-bundle\") pod \"splitter-graph-a019b-69676cb5c4-5td7b\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.479258 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.479237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072639f0-8e17-4af7-925d-c3d62da94102-openshift-service-ca-bundle\") pod \"splitter-graph-a019b-69676cb5c4-5td7b\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.481011 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.480992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072639f0-8e17-4af7-925d-c3d62da94102-proxy-tls\") pod \"splitter-graph-a019b-69676cb5c4-5td7b\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.554289 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.554264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:27.686672 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:27.686502 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b"] Jul 16 18:07:27.689742 ip-10-0-135-249 kubenswrapper[2564]: W0716 18:07:27.689714 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072639f0_8e17_4af7_925d_c3d62da94102.slice/crio-81bf2e6b3019b272674f7ebccd26416a4774f4770ffe20c97e44b1ada4cd8bb0 WatchSource:0}: Error finding container 81bf2e6b3019b272674f7ebccd26416a4774f4770ffe20c97e44b1ada4cd8bb0: Status 404 returned error can't find the container with id 81bf2e6b3019b272674f7ebccd26416a4774f4770ffe20c97e44b1ada4cd8bb0 Jul 16 18:07:28.686118 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:28.686078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" event={"ID":"072639f0-8e17-4af7-925d-c3d62da94102","Type":"ContainerStarted","Data":"4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2"} Jul 16 18:07:28.686504 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:28.686138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" event={"ID":"072639f0-8e17-4af7-925d-c3d62da94102","Type":"ContainerStarted","Data":"81bf2e6b3019b272674f7ebccd26416a4774f4770ffe20c97e44b1ada4cd8bb0"} Jul 16 18:07:28.686504 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:28.686182 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:28.704459 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:28.704412 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podStartSLOduration=1.7043987999999999 podStartE2EDuration="1.7043988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:07:28.703503228 +0000 UTC m=+2026.806607069" watchObservedRunningTime="2026-07-16 18:07:28.7043988 +0000 UTC m=+2026.807502640" Jul 16 18:07:29.313523 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:29.313484 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:34.313480 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:34.313440 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:34.697037 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:34.696948 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:36.725971 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.725936 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerID="14f1c4fe4ad4035335d51b9956410f771bbdbf867e0e300e7ba13ec7cb405b52" exitCode=0 Jul 16 18:07:36.726284 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.726007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" event={"ID":"6a1a1886-b7e2-4b8e-9812-feb2615632a8","Type":"ContainerDied","Data":"14f1c4fe4ad4035335d51b9956410f771bbdbf867e0e300e7ba13ec7cb405b52"} Jul 16 18:07:36.726284 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.726060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" event={"ID":"6a1a1886-b7e2-4b8e-9812-feb2615632a8","Type":"ContainerDied","Data":"6efabfd4eba22fc53167f5bef04a12a63e4c9634038c0aa08dd38899d4f261d6"} Jul 16 18:07:36.726284 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.726076 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6efabfd4eba22fc53167f5bef04a12a63e4c9634038c0aa08dd38899d4f261d6" Jul 16 18:07:36.728208 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.728189 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 18:07:36.860923 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.860808 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1a1886-b7e2-4b8e-9812-feb2615632a8-openshift-service-ca-bundle\") pod \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " Jul 16 18:07:36.860923 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.860906 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1a1886-b7e2-4b8e-9812-feb2615632a8-proxy-tls\") pod \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\" (UID: \"6a1a1886-b7e2-4b8e-9812-feb2615632a8\") " Jul 16 18:07:36.861236 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.861208 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1a1886-b7e2-4b8e-9812-feb2615632a8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6a1a1886-b7e2-4b8e-9812-feb2615632a8" (UID: "6a1a1886-b7e2-4b8e-9812-feb2615632a8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:07:36.862977 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.862950 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1a1886-b7e2-4b8e-9812-feb2615632a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a1a1886-b7e2-4b8e-9812-feb2615632a8" (UID: "6a1a1886-b7e2-4b8e-9812-feb2615632a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:07:36.961610 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.961570 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1a1886-b7e2-4b8e-9812-feb2615632a8-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:07:36.961610 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:36.961602 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1a1886-b7e2-4b8e-9812-feb2615632a8-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:07:37.326989 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:37.326952 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b"] Jul 16 18:07:37.327183 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:37.327161 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" containerID="cri-o://4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2" gracePeriod=30 Jul 16 18:07:37.730851 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:37.730742 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6" Jul 16 18:07:37.751389 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:37.751357 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6"] Jul 16 18:07:37.753727 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:37.753691 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a90bb-5b767cc4d9-qrwl6"] Jul 16 18:07:38.574234 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:38.574198 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" path="/var/lib/kubelet/pods/6a1a1886-b7e2-4b8e-9812-feb2615632a8/volumes" Jul 16 18:07:39.695081 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:39.695037 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:42.580603 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:42.580563 2564 scope.go:117] "RemoveContainer" containerID="14f1c4fe4ad4035335d51b9956410f771bbdbf867e0e300e7ba13ec7cb405b52" Jul 16 18:07:44.695455 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:44.695412 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:49.695273 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:49.695234 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:49.695653 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:49.695343 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:07:54.695342 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:54.695302 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:07:59.695525 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:07:59.695477 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:08:04.695655 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:04.695562 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:08:06.794278 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.794233 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6"] Jul 16 18:08:06.794740 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.794666 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" Jul 16 18:08:06.794740 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.794680 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" Jul 16 18:08:06.794812 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.794766 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1a1886-b7e2-4b8e-9812-feb2615632a8" containerName="sequence-graph-a90bb" Jul 16 18:08:06.797661 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.797638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:06.800229 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.800201 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-97269-serving-cert\"" Jul 16 18:08:06.800376 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.800241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-97269-kube-rbac-proxy-sar-config\"" Jul 16 18:08:06.809198 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.809166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6"] Jul 16 18:08:06.925339 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.925272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9537029c-8eb2-4a36-b628-5b9976805f12-openshift-service-ca-bundle\") pod \"switch-graph-97269-f566d894d-nvnf6\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:06.925521 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:06.925425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9537029c-8eb2-4a36-b628-5b9976805f12-proxy-tls\") pod \"switch-graph-97269-f566d894d-nvnf6\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:07.026184 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.026145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9537029c-8eb2-4a36-b628-5b9976805f12-proxy-tls\") pod \"switch-graph-97269-f566d894d-nvnf6\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:07.026356 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.026191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9537029c-8eb2-4a36-b628-5b9976805f12-openshift-service-ca-bundle\") pod \"switch-graph-97269-f566d894d-nvnf6\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:07.026907 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.026883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9537029c-8eb2-4a36-b628-5b9976805f12-openshift-service-ca-bundle\") pod \"switch-graph-97269-f566d894d-nvnf6\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:07.028514 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.028493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9537029c-8eb2-4a36-b628-5b9976805f12-proxy-tls\") pod \"switch-graph-97269-f566d894d-nvnf6\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:07.108777 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.108670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:07.245733 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.245707 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6"] Jul 16 18:08:07.248541 ip-10-0-135-249 kubenswrapper[2564]: W0716 18:08:07.248503 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9537029c_8eb2_4a36_b628_5b9976805f12.slice/crio-547aadeaa886583744fd11bf02af9dc9f6edeedfced7ad0a3dbc1f897c9f3920 WatchSource:0}: Error finding container 547aadeaa886583744fd11bf02af9dc9f6edeedfced7ad0a3dbc1f897c9f3920: Status 404 returned error can't find the container with id 547aadeaa886583744fd11bf02af9dc9f6edeedfced7ad0a3dbc1f897c9f3920 Jul 16 18:08:07.477567 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.477537 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:08:07.632676 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.632642 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072639f0-8e17-4af7-925d-c3d62da94102-proxy-tls\") pod \"072639f0-8e17-4af7-925d-c3d62da94102\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " Jul 16 18:08:07.632885 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.632686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072639f0-8e17-4af7-925d-c3d62da94102-openshift-service-ca-bundle\") pod \"072639f0-8e17-4af7-925d-c3d62da94102\" (UID: \"072639f0-8e17-4af7-925d-c3d62da94102\") " Jul 16 18:08:07.633091 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.633062 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072639f0-8e17-4af7-925d-c3d62da94102-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "072639f0-8e17-4af7-925d-c3d62da94102" (UID: "072639f0-8e17-4af7-925d-c3d62da94102"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:08:07.634776 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.634753 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072639f0-8e17-4af7-925d-c3d62da94102-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "072639f0-8e17-4af7-925d-c3d62da94102" (UID: "072639f0-8e17-4af7-925d-c3d62da94102"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:08:07.734004 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.733911 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072639f0-8e17-4af7-925d-c3d62da94102-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:08:07.734004 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.733945 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/072639f0-8e17-4af7-925d-c3d62da94102-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:08:07.866481 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.866442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" event={"ID":"9537029c-8eb2-4a36-b628-5b9976805f12","Type":"ContainerStarted","Data":"ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7"} Jul 16 18:08:07.866996 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.866605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:07.866996 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.866631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" event={"ID":"9537029c-8eb2-4a36-b628-5b9976805f12","Type":"ContainerStarted","Data":"547aadeaa886583744fd11bf02af9dc9f6edeedfced7ad0a3dbc1f897c9f3920"} Jul 16 18:08:07.867711 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.867682 2564 generic.go:356] "Generic (PLEG): container finished" podID="072639f0-8e17-4af7-925d-c3d62da94102" containerID="4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2" exitCode=0 Jul 16 18:08:07.867799 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.867725 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" Jul 16 18:08:07.867799 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.867756 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" event={"ID":"072639f0-8e17-4af7-925d-c3d62da94102","Type":"ContainerDied","Data":"4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2"} Jul 16 18:08:07.867916 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.867800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b" event={"ID":"072639f0-8e17-4af7-925d-c3d62da94102","Type":"ContainerDied","Data":"81bf2e6b3019b272674f7ebccd26416a4774f4770ffe20c97e44b1ada4cd8bb0"} Jul 16 18:08:07.867916 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.867845 2564 scope.go:117] "RemoveContainer" containerID="4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2" Jul 16 18:08:07.879537 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.879510 2564 scope.go:117] "RemoveContainer" containerID="4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2" Jul 16 18:08:07.880042 ip-10-0-135-249 kubenswrapper[2564]: E0716 18:08:07.880013 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2\": container with ID starting with 4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2 not found: ID does not exist" containerID="4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2" Jul 16 18:08:07.880229 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.880051 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2"} err="failed to get container status \"4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2\": rpc error: code = NotFound desc = could not find container \"4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2\": container with ID starting with 4ae6e7f633dd0d067449fb3abb30d240b3ffe43f7950b3c5b2466dd235d155f2 not found: ID does not exist" Jul 16 18:08:07.883949 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.883906 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podStartSLOduration=1.8838920319999999 podStartE2EDuration="1.883892032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:08:07.881944771 +0000 UTC m=+2065.985048612" watchObservedRunningTime="2026-07-16 18:08:07.883892032 +0000 UTC m=+2065.986995902" Jul 16 18:08:07.897695 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.897666 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b"] Jul 16 18:08:07.901972 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:07.901946 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a019b-69676cb5c4-5td7b"] Jul 16 18:08:08.575162 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:08.575127 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072639f0-8e17-4af7-925d-c3d62da94102" path="/var/lib/kubelet/pods/072639f0-8e17-4af7-925d-c3d62da94102/volumes" Jul 16 18:08:13.880950 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:13.880918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:08:37.599679 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.599634 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq"] Jul 16 18:08:37.600237 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.600191 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" Jul 16 18:08:37.600237 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.600212 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" Jul 16 18:08:37.600362 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.600343 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="072639f0-8e17-4af7-925d-c3d62da94102" containerName="splitter-graph-a019b" Jul 16 18:08:37.603279 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.603259 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:37.605891 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.605874 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-21cc5-serving-cert\"" Jul 16 18:08:37.605891 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.605879 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-21cc5-kube-rbac-proxy-sar-config\"" Jul 16 18:08:37.614146 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.614120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq"] Jul 16 18:08:37.711242 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.711205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be09695-24b0-46d7-92f5-bc651a5e0d9d-openshift-service-ca-bundle\") pod \"splitter-graph-21cc5-6dc7d7677b-phnvq\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:37.711401 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.711305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be09695-24b0-46d7-92f5-bc651a5e0d9d-proxy-tls\") pod \"splitter-graph-21cc5-6dc7d7677b-phnvq\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:37.811951 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.811916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be09695-24b0-46d7-92f5-bc651a5e0d9d-openshift-service-ca-bundle\") pod \"splitter-graph-21cc5-6dc7d7677b-phnvq\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:37.812117 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.811983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be09695-24b0-46d7-92f5-bc651a5e0d9d-proxy-tls\") pod \"splitter-graph-21cc5-6dc7d7677b-phnvq\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:37.812561 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.812540 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be09695-24b0-46d7-92f5-bc651a5e0d9d-openshift-service-ca-bundle\") pod \"splitter-graph-21cc5-6dc7d7677b-phnvq\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:37.814424 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.814401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be09695-24b0-46d7-92f5-bc651a5e0d9d-proxy-tls\") pod \"splitter-graph-21cc5-6dc7d7677b-phnvq\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:37.913399 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:37.913310 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:38.047944 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:38.047921 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq"] Jul 16 18:08:38.049857 ip-10-0-135-249 kubenswrapper[2564]: W0716 18:08:38.049799 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be09695_24b0_46d7_92f5_bc651a5e0d9d.slice/crio-c57b18485a5d9f4bce0ddd1bca2dadc4c6654d2f171332842626f4bced835b81 WatchSource:0}: Error finding container c57b18485a5d9f4bce0ddd1bca2dadc4c6654d2f171332842626f4bced835b81: Status 404 returned error can't find the container with id c57b18485a5d9f4bce0ddd1bca2dadc4c6654d2f171332842626f4bced835b81 Jul 16 18:08:39.021338 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:39.021302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" event={"ID":"3be09695-24b0-46d7-92f5-bc651a5e0d9d","Type":"ContainerStarted","Data":"e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2"} Jul 16 18:08:39.021846 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:39.021360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" event={"ID":"3be09695-24b0-46d7-92f5-bc651a5e0d9d","Type":"ContainerStarted","Data":"c57b18485a5d9f4bce0ddd1bca2dadc4c6654d2f171332842626f4bced835b81"} Jul 16 18:08:39.021846 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:39.021441 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:08:39.038775 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:39.038723 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podStartSLOduration=2.038708694 podStartE2EDuration="2.038708694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:08:39.037191536 +0000 UTC m=+2097.140295377" watchObservedRunningTime="2026-07-16 18:08:39.038708694 +0000 UTC m=+2097.141812604" Jul 16 18:08:42.494524 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:42.494493 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:08:42.499855 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:42.499805 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:08:45.031759 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:08:45.031729 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:13:42.522248 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:13:42.522215 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:13:42.529439 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:13:42.529414 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:16:52.274294 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:16:52.274259 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq"] Jul 16 18:16:52.274899 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:16:52.274502 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" containerID="cri-o://e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2" gracePeriod=30 Jul 16 18:16:55.030018 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:16:55.029978 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:17:00.029509 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:00.029460 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:17:05.029719 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:05.029620 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:17:05.030133 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:05.029745 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:17:10.029929 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:10.029876 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:17:15.030523 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:15.030475 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:17:20.030379 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:20.030338 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:17:22.423320 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.423293 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:17:22.457718 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.457683 2564 generic.go:356] "Generic (PLEG): container finished" podID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerID="e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2" exitCode=0 Jul 16 18:17:22.457915 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.457739 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" Jul 16 18:17:22.457915 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.457742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" event={"ID":"3be09695-24b0-46d7-92f5-bc651a5e0d9d","Type":"ContainerDied","Data":"e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2"} Jul 16 18:17:22.457915 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.457886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq" event={"ID":"3be09695-24b0-46d7-92f5-bc651a5e0d9d","Type":"ContainerDied","Data":"c57b18485a5d9f4bce0ddd1bca2dadc4c6654d2f171332842626f4bced835b81"} Jul 16 18:17:22.457915 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.457911 2564 scope.go:117] "RemoveContainer" containerID="e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2" Jul 16 18:17:22.467509 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.467488 2564 scope.go:117] "RemoveContainer" containerID="e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2" Jul 16 18:17:22.467818 ip-10-0-135-249 kubenswrapper[2564]: E0716 18:17:22.467762 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2\": container with ID starting with e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2 not found: ID does not exist" containerID="e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2" Jul 16 18:17:22.467818 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.467792 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2"} err="failed to get container status \"e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2\": rpc error: code = NotFound desc = could not find container \"e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2\": container with ID starting with e37182091def243cc50fafba8225a436fa952b6a663c15b4a2c8bbe784ff1be2 not found: ID does not exist" Jul 16 18:17:22.540269 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.540168 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be09695-24b0-46d7-92f5-bc651a5e0d9d-proxy-tls\") pod \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " Jul 16 18:17:22.540269 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.540252 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be09695-24b0-46d7-92f5-bc651a5e0d9d-openshift-service-ca-bundle\") pod \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\" (UID: \"3be09695-24b0-46d7-92f5-bc651a5e0d9d\") " Jul 16 18:17:22.540616 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.540592 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be09695-24b0-46d7-92f5-bc651a5e0d9d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3be09695-24b0-46d7-92f5-bc651a5e0d9d" (UID: "3be09695-24b0-46d7-92f5-bc651a5e0d9d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:17:22.542340 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.542318 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be09695-24b0-46d7-92f5-bc651a5e0d9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3be09695-24b0-46d7-92f5-bc651a5e0d9d" (UID: "3be09695-24b0-46d7-92f5-bc651a5e0d9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:17:22.641401 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.641370 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be09695-24b0-46d7-92f5-bc651a5e0d9d-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:17:22.641401 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.641396 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be09695-24b0-46d7-92f5-bc651a5e0d9d-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:17:22.773228 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.773194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq"] Jul 16 18:17:22.777477 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:22.777452 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-21cc5-6dc7d7677b-phnvq"] Jul 16 18:17:24.575080 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:17:24.575038 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" path="/var/lib/kubelet/pods/3be09695-24b0-46d7-92f5-bc651a5e0d9d/volumes" Jul 16 18:18:42.551206 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:18:42.551129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:18:42.558737 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:18:42.558716 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:23:42.581441 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:23:42.581360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:23:42.589686 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:23:42.589661 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:24:25.928335 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:25.928274 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6"] Jul 16 18:24:25.928880 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:25.928557 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" containerID="cri-o://ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7" gracePeriod=30 Jul 16 18:24:28.878823 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:28.878783 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:24:33.879157 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:33.879115 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:24:38.878324 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:38.878277 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:24:38.878697 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:38.878384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:24:43.787379 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:43.787330 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:43.878369 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:43.878327 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:24:44.658808 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:44.658778 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:45.550912 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:45.550881 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:46.401094 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:46.401063 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:47.259968 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:47.259930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:48.092837 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:48.092792 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:48.878518 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:48.878472 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:24:48.921188 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:48.921148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:49.758981 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:49.758945 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:50.622141 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:50.622107 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:51.474183 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:51.474148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:52.312456 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:52.312418 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:53.136983 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:53.136951 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:53.878817 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:53.878777 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 18:24:53.956378 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:53.956341 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:54.837598 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:54.837561 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-97269-f566d894d-nvnf6_9537029c-8eb2-4a36-b628-5b9976805f12/switch-graph-97269/0.log" Jul 16 18:24:56.074858 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.074812 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:24:56.225227 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.225143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9537029c-8eb2-4a36-b628-5b9976805f12-proxy-tls\") pod \"9537029c-8eb2-4a36-b628-5b9976805f12\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " Jul 16 18:24:56.225374 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.225257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9537029c-8eb2-4a36-b628-5b9976805f12-openshift-service-ca-bundle\") pod \"9537029c-8eb2-4a36-b628-5b9976805f12\" (UID: \"9537029c-8eb2-4a36-b628-5b9976805f12\") " Jul 16 18:24:56.225641 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.225610 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9537029c-8eb2-4a36-b628-5b9976805f12-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9537029c-8eb2-4a36-b628-5b9976805f12" (UID: "9537029c-8eb2-4a36-b628-5b9976805f12"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 18:24:56.227138 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.227110 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9537029c-8eb2-4a36-b628-5b9976805f12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9537029c-8eb2-4a36-b628-5b9976805f12" (UID: "9537029c-8eb2-4a36-b628-5b9976805f12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 18:24:56.326414 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.326377 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9537029c-8eb2-4a36-b628-5b9976805f12-openshift-service-ca-bundle\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:24:56.326414 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.326407 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9537029c-8eb2-4a36-b628-5b9976805f12-proxy-tls\") on node \"ip-10-0-135-249.ec2.internal\" DevicePath \"\"" Jul 16 18:24:56.544035 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.544003 2564 generic.go:356] "Generic (PLEG): container finished" podID="9537029c-8eb2-4a36-b628-5b9976805f12" containerID="ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7" exitCode=0 Jul 16 18:24:56.544202 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.544044 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" Jul 16 18:24:56.544202 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.544049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" event={"ID":"9537029c-8eb2-4a36-b628-5b9976805f12","Type":"ContainerDied","Data":"ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7"} Jul 16 18:24:56.544202 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.544153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6" event={"ID":"9537029c-8eb2-4a36-b628-5b9976805f12","Type":"ContainerDied","Data":"547aadeaa886583744fd11bf02af9dc9f6edeedfced7ad0a3dbc1f897c9f3920"} Jul 16 18:24:56.544202 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.544177 2564 scope.go:117] "RemoveContainer" containerID="ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7" Jul 16 18:24:56.552779 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.552764 2564 scope.go:117] "RemoveContainer" containerID="ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7" Jul 16 18:24:56.553040 ip-10-0-135-249 kubenswrapper[2564]: E0716 18:24:56.553022 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7\": container with ID starting with ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7 not found: ID does not exist" containerID="ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7" Jul 16 18:24:56.553104 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.553047 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7"} err="failed to get container status \"ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7\": rpc error: code = NotFound desc = could not find container \"ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7\": container with ID starting with ab235d8843acadde568fe6cd77691547155b2b31ab4d17383de450d3219158f7 not found: ID does not exist" Jul 16 18:24:56.568354 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.568328 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6"] Jul 16 18:24:56.574773 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:56.574751 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-97269-f566d894d-nvnf6"] Jul 16 18:24:58.573448 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:24:58.573415 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" path="/var/lib/kubelet/pods/9537029c-8eb2-4a36-b628-5b9976805f12/volumes" Jul 16 18:25:02.069842 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:02.069809 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kq8gb_50cdddea-617e-4f65-acb6-927e443845ff/global-pull-secret-syncer/0.log" Jul 16 18:25:02.186130 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:02.186095 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rnkws_cedc1c45-ce74-4f57-95ba-77eabe10b456/konnectivity-agent/0.log" Jul 16 18:25:02.211142 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:02.211107 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-249.ec2.internal_be2e4781f1bc5622abf5feee16446d86/haproxy/0.log" Jul 16 18:25:06.302206 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.302167 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df435d78-d969-405c-b381-128a788e2fb1/alertmanager/0.log" Jul 16 18:25:06.326226 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.326194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df435d78-d969-405c-b381-128a788e2fb1/config-reloader/0.log" Jul 16 18:25:06.351273 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.351244 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df435d78-d969-405c-b381-128a788e2fb1/kube-rbac-proxy-web/0.log" Jul 16 18:25:06.381272 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.381233 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df435d78-d969-405c-b381-128a788e2fb1/kube-rbac-proxy/0.log" Jul 16 18:25:06.406304 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.406273 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df435d78-d969-405c-b381-128a788e2fb1/kube-rbac-proxy-metric/0.log" Jul 16 18:25:06.429715 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.429686 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df435d78-d969-405c-b381-128a788e2fb1/prom-label-proxy/0.log" Jul 16 18:25:06.454211 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.454181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df435d78-d969-405c-b381-128a788e2fb1/init-config-reloader/0.log" Jul 16 18:25:06.623146 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.623064 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8f844764d-fwzz5_e2c9404d-97d8-4779-bf17-7d19fffad34d/metrics-server/0.log" Jul 16 18:25:06.775586 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.775559 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkx7z_03b6d4fd-b9c1-404f-97a9-86bd2753f783/node-exporter/0.log" Jul 16 18:25:06.798652 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.798596 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkx7z_03b6d4fd-b9c1-404f-97a9-86bd2753f783/kube-rbac-proxy/0.log" Jul 16 18:25:06.830343 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:06.830314 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkx7z_03b6d4fd-b9c1-404f-97a9-86bd2753f783/init-textfile/0.log" Jul 16 18:25:07.269025 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.268993 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-vp4tk_289decec-8dcd-47b6-8dbe-e7f27e8eb881/prometheus-operator/0.log" Jul 16 18:25:07.290602 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.290573 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-vp4tk_289decec-8dcd-47b6-8dbe-e7f27e8eb881/kube-rbac-proxy/0.log" Jul 16 18:25:07.322715 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.322688 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-5c85t_1f2d0e39-8681-4c1c-9cf2-739835451b6b/prometheus-operator-admission-webhook/0.log" Jul 16 18:25:07.368434 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.368403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db58cfb4c-zddth_a2deeee9-5fc8-4247-9a8f-af5356184fa9/telemeter-client/0.log" Jul 16 18:25:07.402626 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.402596 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db58cfb4c-zddth_a2deeee9-5fc8-4247-9a8f-af5356184fa9/reload/0.log" Jul 16 18:25:07.437282 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.437252 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db58cfb4c-zddth_a2deeee9-5fc8-4247-9a8f-af5356184fa9/kube-rbac-proxy/0.log" Jul 16 18:25:07.490193 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.490164 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8595f96b5c-n5lhx_dfd5e33b-3b9f-40ea-9aef-50c4867e68e2/thanos-query/0.log" Jul 16 18:25:07.521142 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.521059 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8595f96b5c-n5lhx_dfd5e33b-3b9f-40ea-9aef-50c4867e68e2/kube-rbac-proxy-web/0.log" Jul 16 18:25:07.551359 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.551328 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8595f96b5c-n5lhx_dfd5e33b-3b9f-40ea-9aef-50c4867e68e2/kube-rbac-proxy/0.log" Jul 16 18:25:07.585227 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.585197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8595f96b5c-n5lhx_dfd5e33b-3b9f-40ea-9aef-50c4867e68e2/prom-label-proxy/0.log" Jul 16 18:25:07.611608 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.611581 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8595f96b5c-n5lhx_dfd5e33b-3b9f-40ea-9aef-50c4867e68e2/kube-rbac-proxy-rules/0.log" Jul 16 18:25:07.638147 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:07.638123 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8595f96b5c-n5lhx_dfd5e33b-3b9f-40ea-9aef-50c4867e68e2/kube-rbac-proxy-metrics/0.log" Jul 16 18:25:09.657990 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.657951 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2"] Jul 16 18:25:09.658345 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.658291 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" Jul 16 18:25:09.658345 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.658302 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" Jul 16 18:25:09.658345 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.658318 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" Jul 16 18:25:09.658345 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.658325 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" Jul 16 18:25:09.658474 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.658378 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9537029c-8eb2-4a36-b628-5b9976805f12" containerName="switch-graph-97269" Jul 16 18:25:09.658474 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.658387 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be09695-24b0-46d7-92f5-bc651a5e0d9d" containerName="splitter-graph-21cc5" Jul 16 18:25:09.662556 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.662536 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.664674 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.664653 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpk2v\"/\"kube-root-ca.crt\"" Jul 16 18:25:09.664882 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.664653 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hpk2v\"/\"default-dockercfg-g2x2f\"" Jul 16 18:25:09.665445 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.665426 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpk2v\"/\"openshift-service-ca.crt\"" Jul 16 18:25:09.669724 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.669644 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2"] Jul 16 18:25:09.679625 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.679606 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cdc4d869f-kllzd_b8f3ae94-de86-49cb-aeff-3a0e7cd9c28f/console/0.log" Jul 16 18:25:09.729735 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.729708 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-tj4nn_04b6d468-7be6-43da-ba89-88cde195a8f3/download-server/0.log" Jul 16 18:25:09.845238 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.845194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-sys\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.845427 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.845254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-proc\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.845427 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.845325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2dg\" (UniqueName: \"kubernetes.io/projected/ba971fd3-b3c9-43f2-ae48-800b34b085c8-kube-api-access-zm2dg\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.845427 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.845395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-podres\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.845545 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.845445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-lib-modules\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946411 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2dg\" (UniqueName: \"kubernetes.io/projected/ba971fd3-b3c9-43f2-ae48-800b34b085c8-kube-api-access-zm2dg\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946411 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-podres\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946610 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-lib-modules\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946610 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-podres\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946712 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-sys\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946712 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-lib-modules\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946712 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-sys\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946712 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-proc\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.946875 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.946775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ba971fd3-b3c9-43f2-ae48-800b34b085c8-proc\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.955705 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.955683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2dg\" (UniqueName: \"kubernetes.io/projected/ba971fd3-b3c9-43f2-ae48-800b34b085c8-kube-api-access-zm2dg\") pod \"perf-node-gather-daemonset-sc8s2\" (UID: \"ba971fd3-b3c9-43f2-ae48-800b34b085c8\") " pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:09.973628 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:09.973603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:10.102866 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:10.102821 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2"] Jul 16 18:25:10.105233 ip-10-0-135-249 kubenswrapper[2564]: W0716 18:25:10.105207 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podba971fd3_b3c9_43f2_ae48_800b34b085c8.slice/crio-0d47b9e10763b2d8f2f359fb93b939be772d968bb12e90dbf727c3499a0bd66d WatchSource:0}: Error finding container 0d47b9e10763b2d8f2f359fb93b939be772d968bb12e90dbf727c3499a0bd66d: Status 404 returned error can't find the container with id 0d47b9e10763b2d8f2f359fb93b939be772d968bb12e90dbf727c3499a0bd66d Jul 16 18:25:10.608913 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:10.608878 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" event={"ID":"ba971fd3-b3c9-43f2-ae48-800b34b085c8","Type":"ContainerStarted","Data":"81946d7ee5727288ae44927dc08e2f9959b2c73018af008b1773fd093743d1a3"} Jul 16 18:25:10.609114 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:10.608931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" event={"ID":"ba971fd3-b3c9-43f2-ae48-800b34b085c8","Type":"ContainerStarted","Data":"0d47b9e10763b2d8f2f359fb93b939be772d968bb12e90dbf727c3499a0bd66d"} Jul 16 18:25:10.609114 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:10.609062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:10.627246 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:10.627197 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" podStartSLOduration=1.6271837150000001 podStartE2EDuration="1.627183715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:25:10.624988816 +0000 UTC m=+3088.728092654" watchObservedRunningTime="2026-07-16 18:25:10.627183715 +0000 UTC m=+3088.730287553" Jul 16 18:25:11.332212 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:11.332173 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z54xl_41658552-fcc5-420f-80c1-9078dfb115a6/dns/0.log" Jul 16 18:25:11.369452 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:11.369424 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z54xl_41658552-fcc5-420f-80c1-9078dfb115a6/kube-rbac-proxy/0.log" Jul 16 18:25:11.467376 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:11.467345 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wkxtr_b95008c8-7da1-4f94-93dc-4d871db2edc2/dns-node-resolver/0.log" Jul 16 18:25:11.956464 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:11.956435 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6565964545-dhkr2_1fa1250f-3079-4f3e-a981-515e8ec72ae8/registry/0.log" Jul 16 18:25:12.034235 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:12.034197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lfccv_a620fd44-37b6-4caa-8ecd-e2e785fff8bd/node-ca/0.log" Jul 16 18:25:12.790972 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:12.790940 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78d54cf4db-8dh9v_86262674-0c27-49a1-89a3-01071d6fbdaf/router/0.log" Jul 16 18:25:13.136639 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:13.136558 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-22r9v_0fb5d45c-4ace-43e5-9420-744cb4d952d2/serve-healthcheck-canary/0.log" Jul 16 18:25:13.550696 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:13.550662 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-dhckd_80e75ef9-a0a6-4037-8db0-3cd0c08369c6/insights-operator/0.log" Jul 16 18:25:13.552752 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:13.552723 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-dhckd_80e75ef9-a0a6-4037-8db0-3cd0c08369c6/insights-operator/1.log" Jul 16 18:25:13.575558 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:13.575531 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gfv6q_98e95ef5-11f0-459f-8722-0d796d9b78d3/kube-rbac-proxy/0.log" Jul 16 18:25:13.596616 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:13.596591 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gfv6q_98e95ef5-11f0-459f-8722-0d796d9b78d3/exporter/0.log" Jul 16 18:25:13.618077 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:13.618048 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gfv6q_98e95ef5-11f0-459f-8722-0d796d9b78d3/extractor/0.log" Jul 16 18:25:15.802203 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:15.802167 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-24tcj_7098ba81-2f72-44d6-ae62-75d1a5fb5fbb/manager/0.log" Jul 16 18:25:15.982661 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:15.982625 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-b7sxm_0405a103-1e74-40fc-946b-119e085a9c71/manager/0.log" Jul 16 18:25:16.062012 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:16.061931 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-vvjr7_05f3e46f-b716-48b1-825e-6cc8c5e17516/seaweedfs/0.log" Jul 16 18:25:16.624063 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:16.624038 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hpk2v/perf-node-gather-daemonset-sc8s2" Jul 16 18:25:21.565652 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:21.565620 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2g4c9_69eb91ad-a8ca-470a-9424-6c3d2dbb9320/kube-multus-additional-cni-plugins/0.log" Jul 16 18:25:21.587958 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:21.587930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2g4c9_69eb91ad-a8ca-470a-9424-6c3d2dbb9320/egress-router-binary-copy/0.log" Jul 16 18:25:21.611039 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:21.611013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2g4c9_69eb91ad-a8ca-470a-9424-6c3d2dbb9320/cni-plugins/0.log" Jul 16 18:25:21.633710 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:21.633687 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2g4c9_69eb91ad-a8ca-470a-9424-6c3d2dbb9320/bond-cni-plugin/0.log" Jul 16 18:25:21.657743 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:21.657709 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2g4c9_69eb91ad-a8ca-470a-9424-6c3d2dbb9320/routeoverride-cni/0.log" Jul 16 18:25:21.678853 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:21.678814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2g4c9_69eb91ad-a8ca-470a-9424-6c3d2dbb9320/whereabouts-cni-bincopy/0.log" Jul 16 18:25:21.701660 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:21.701629 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2g4c9_69eb91ad-a8ca-470a-9424-6c3d2dbb9320/whereabouts-cni/0.log" Jul 16 18:25:22.119789 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:22.119745 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-km6mf_9f855705-abda-416d-b6a8-61b67eaf3684/kube-multus/0.log" Jul 16 18:25:22.246631 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:22.246602 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-87b84_3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc/network-metrics-daemon/0.log" Jul 16 18:25:22.265011 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:22.264984 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-87b84_3e61ff64-2ff1-4c96-ae1a-ad22e9fdddcc/kube-rbac-proxy/0.log" Jul 16 18:25:23.540708 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.540676 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-controller/0.log" Jul 16 18:25:23.560616 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.560587 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/1.log" Jul 16 18:25:23.560616 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.560607 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovn-acl-logging/0.log" Jul 16 18:25:23.581487 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.581460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/kube-rbac-proxy-node/0.log" Jul 16 18:25:23.604814 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.604790 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/kube-rbac-proxy-ovn-metrics/0.log" Jul 16 18:25:23.623084 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.623062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/northd/0.log" Jul 16 18:25:23.644255 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.644230 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/nbdb/0.log" Jul 16 18:25:23.664290 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.664263 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/sbdb/0.log" Jul 16 18:25:23.728027 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:23.727975 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfz5t_28db96b1-887b-4387-b604-7eea68068a4b/ovnkube-controller/0.log" Jul 16 18:25:24.684218 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:24.684189 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-kc2td_acce4b1f-453f-41ff-939b-240d335957e3/check-endpoints/0.log" Jul 16 18:25:24.730450 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:24.730425 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bv96g_76540f10-29f6-4fe6-be47-760ea023e036/network-check-target-container/0.log" Jul 16 18:25:25.710765 ip-10-0-135-249 kubenswrapper[2564]: I0716 18:25:25.710679 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-glk67_d34db377-5cb5-48a2-bf2d-24892a1cc619/iptables-alerter/0.log"