Jul 27 17:28:02.443541 ip-10-0-131-71 systemd[1]: Starting Kubernetes Kubelet... Jul 27 17:28:02.798636 ip-10-0-131-71 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 17:28:02.798636 ip-10-0-131-71 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 17:28:02.798636 ip-10-0-131-71 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 17:28:02.798636 ip-10-0-131-71 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 17:28:02.798636 ip-10-0-131-71 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 17:28:02.800392 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.800305 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 17:28:02.804864 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804849 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804865 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804869 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804872 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804875 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804878 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804881 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804884 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804887 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804889 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804893 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804895 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804902 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804905 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.804900 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804908 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804910 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804913 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804916 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804919 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804922 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804924 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804927 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804930 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804932 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804935 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804937 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804940 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804943 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804945 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804948 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804950 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804953 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804957 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804961 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.805232 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804963 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804969 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804973 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804976 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804979 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804982 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804985 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804988 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804991 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804994 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804996 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.804999 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805002 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805005 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805009 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805012 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805015 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805018 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805021 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.805714 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805024 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805027 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805029 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805032 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805035 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805038 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805040 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805043 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805045 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805048 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805051 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805053 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805056 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805058 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805061 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805063 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805066 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805069 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805072 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805074 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.806213 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805077 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805080 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805082 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805085 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805087 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805090 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805093 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805096 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805098 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805101 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805105 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805108 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805110 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805113 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805116 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805118 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805121 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805124 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805126 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805130 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.806960 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805134 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805136 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805139 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805142 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805145 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805147 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805847 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805853 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805856 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805859 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805862 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805865 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805868 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805871 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805874 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805876 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805879 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805882 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805885 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805888 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.807755 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805891 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805894 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805897 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805899 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805902 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805905 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805908 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805910 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805913 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805915 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805918 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805921 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805923 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805926 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805928 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805931 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805933 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805936 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805938 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805941 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.808260 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805944 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805946 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805949 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805952 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805954 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805957 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805961 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805963 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805966 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805971 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805975 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805979 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805982 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805986 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805989 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805992 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805996 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.805999 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806002 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.808757 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806005 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806008 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806010 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806013 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806016 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806019 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806022 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806025 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806027 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806030 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806032 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806035 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806037 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806040 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806044 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806047 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806049 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806052 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806055 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806057 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.809226 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806060 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806062 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806065 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806068 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806071 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806074 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806078 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806081 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806084 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806086 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806089 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806092 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806094 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806097 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806099 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806102 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806105 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806107 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806110 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.809763 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806112 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806115 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806117 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806120 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806123 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806125 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.806128 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807901 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807912 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807916 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807919 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807922 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807925 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807931 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807936 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807939 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807942 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807945 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807948 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807951 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.810224 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807954 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807957 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807959 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807962 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807965 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807967 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807970 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807973 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807976 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807978 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807981 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807983 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807986 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807988 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807991 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807994 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807996 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.807999 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808001 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808005 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.810726 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808008 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808010 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808014 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808018 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808021 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808023 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808026 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808029 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808031 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808034 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808036 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808039 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808042 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808044 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808047 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808050 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808052 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808055 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808058 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.811210 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808061 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808064 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808066 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808069 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808071 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808075 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808077 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808080 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808082 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808085 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808087 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808090 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808093 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808096 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808099 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808101 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808104 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808107 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808109 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.811698 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808113 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808116 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808119 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808122 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808125 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808128 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808130 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808133 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808135 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808138 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808140 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808143 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808145 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808147 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808150 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808153 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808155 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808158 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808161 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808163 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.812148 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808167 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808170 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808173 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808175 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808178 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808181 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808184 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808187 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808278 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808286 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808293 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808302 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808306 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808310 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808313 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808317 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808320 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808323 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808327 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808330 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808333 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808336 2564 flags.go:64] FLAG: --cgroup-root="" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808339 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 17:28:02.812658 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808342 2564 flags.go:64] FLAG: --client-ca-file="" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808346 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808349 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808354 2564 flags.go:64] FLAG: --cluster-domain="" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808357 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808359 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808362 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808366 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808370 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808372 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808375 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808378 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808381 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808384 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808390 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808395 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808398 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808402 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808405 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808408 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808411 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808413 2564 flags.go:64] FLAG: --enable-server="true" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808416 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808421 2564 flags.go:64] FLAG: --event-burst="100" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808424 2564 flags.go:64] FLAG: --event-qps="50" Jul 27 17:28:02.813207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808426 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808429 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808432 2564 flags.go:64] FLAG: --eviction-hard="" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808436 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808439 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808442 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808445 2564 flags.go:64] FLAG: --eviction-soft="" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808448 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808451 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808454 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808457 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808460 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808463 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808466 2564 flags.go:64] FLAG: --feature-gates="" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808469 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808472 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808475 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808478 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808481 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808484 2564 flags.go:64] FLAG: --help="false" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808487 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-131-71.ec2.internal" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808491 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808495 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 17:28:02.813829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808498 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808502 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808505 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808508 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808511 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808514 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808516 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808519 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808522 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808525 2564 flags.go:64] FLAG: --kube-reserved="" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808528 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808530 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808533 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808536 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808539 2564 flags.go:64] FLAG: --lock-file="" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808542 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808545 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808548 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808552 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808557 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808560 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808562 2564 flags.go:64] FLAG: --logging-format="text" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808565 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808569 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 17:28:02.814415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808571 2564 flags.go:64] FLAG: --manifest-url="" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808574 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808578 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808581 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808585 2564 flags.go:64] FLAG: --max-pods="110" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808588 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808593 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808597 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808600 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808603 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808606 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808609 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808616 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808619 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808622 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808625 2564 flags.go:64] FLAG: --pod-cidr="" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808628 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808633 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808636 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808639 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808642 2564 flags.go:64] FLAG: --port="10250" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808645 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808648 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b46e701f258b75f4" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808651 2564 flags.go:64] FLAG: --qos-reserved="" Jul 27 17:28:02.815008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808654 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808657 2564 flags.go:64] FLAG: --register-node="true" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808660 2564 flags.go:64] FLAG: --register-with-taints="" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808663 2564 flags.go:64] FLAG: --registry-burst="10" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808667 2564 flags.go:64] FLAG: --registry-qps="5" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808670 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808672 2564 flags.go:64] FLAG: --reserved-memory="" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808676 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808679 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808682 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808685 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808688 2564 flags.go:64] FLAG: --runonce="false" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808691 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808694 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808696 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808701 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808704 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808707 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808710 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808713 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808716 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808719 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808722 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808725 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808728 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 17:28:02.815632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808731 2564 flags.go:64] FLAG: --system-cgroups="" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808734 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808739 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808741 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808744 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808749 2564 flags.go:64] FLAG: --tls-min-version="" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808752 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808754 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808757 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808760 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808763 2564 flags.go:64] FLAG: --v="2" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808767 2564 flags.go:64] FLAG: --version="false" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808771 2564 flags.go:64] FLAG: --vmodule="" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808775 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.808778 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808890 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808893 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808897 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808900 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808902 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808905 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808908 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808913 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.816254 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808916 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808918 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808921 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808924 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808926 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808929 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808931 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808934 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808937 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808939 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808943 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808947 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808950 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808953 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808956 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808959 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808961 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808964 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808966 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.816824 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808969 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808971 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808974 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808977 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808980 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808983 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808985 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808987 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808990 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808993 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808995 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.808997 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809003 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809005 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809008 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809011 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809013 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809016 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809019 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809021 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.817346 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809023 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809026 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809029 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809031 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809034 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809036 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809039 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809041 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809044 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809047 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809049 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809052 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809056 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809060 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809062 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809065 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809068 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809071 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809073 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.817825 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809076 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809078 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809081 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809083 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809086 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809091 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809093 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809096 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809099 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809101 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809104 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809107 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809109 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809112 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809114 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809117 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809120 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809122 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809125 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809128 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.818290 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809130 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809133 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809136 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809138 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809141 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809143 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809146 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809149 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809153 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809156 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809160 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809163 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.809166 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.809635 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.816550 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 17:28:02.818814 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.816565 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.817308 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.817315 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817370 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817375 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817379 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817382 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817385 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817388 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817391 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817393 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817396 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817399 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817401 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817404 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817407 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817410 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817412 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817415 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817418 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.819195 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817420 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817423 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817427 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817430 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817433 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817436 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817438 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817442 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817445 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817448 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817450 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817453 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817457 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817460 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817462 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817465 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817467 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817470 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817473 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.819708 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817475 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817478 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817481 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817483 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817486 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817488 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817491 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817493 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817496 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817498 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817501 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817504 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817506 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817509 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817511 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817514 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817516 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817519 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817522 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.820158 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817524 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817527 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817531 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817533 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817536 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817538 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817541 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817544 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817547 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817549 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817552 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817554 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817556 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817560 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817563 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817566 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817569 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817572 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817575 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817578 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.820735 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817580 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817583 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817586 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817589 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817592 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817595 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817598 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817601 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817604 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817607 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817609 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817612 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817615 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817618 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817620 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817623 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817626 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817629 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817632 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.821219 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817636 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817639 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817642 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817645 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817647 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.817652 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817760 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817765 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817768 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817771 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817774 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817777 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817780 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817783 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817786 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.821697 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817789 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817791 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817794 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817796 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817799 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817801 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817804 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817806 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817809 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817812 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817814 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817817 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817819 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817822 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817825 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817828 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817831 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817834 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817837 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817840 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.822126 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817842 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817845 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817847 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817850 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817852 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817855 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817858 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817862 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817865 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817869 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817871 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817874 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817877 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817879 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817882 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817886 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817889 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817891 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.822615 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817894 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817897 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817899 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817902 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817904 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817907 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817909 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817912 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817915 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817918 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817920 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817923 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817926 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817928 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817931 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817933 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817936 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817938 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817941 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817943 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.823059 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817945 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817948 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817950 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817952 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817955 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817957 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817960 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817962 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817966 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817970 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817973 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817976 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817978 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817981 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817983 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817986 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817989 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817991 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817994 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817996 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.823558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.817999 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818001 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818004 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818007 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818009 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818012 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818014 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818017 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818019 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818022 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818024 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:02.818027 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.818031 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.818135 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.820351 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 17:28:02.824054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.821682 2564 server.go:1019] "Starting client certificate rotation" Jul 27 17:28:02.824443 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.821784 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 17:28:02.824443 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.821815 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 17:28:02.824443 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.824009 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 17:28:02.828208 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.828189 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 17:28:02.841485 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.841457 2564 log.go:25] "Validated CRI v1 runtime API" Jul 27 17:28:02.844369 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.844353 2564 log.go:25] "Validated CRI v1 image API" Jul 27 17:28:02.845250 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.845221 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 17:28:02.847532 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.847515 2564 fs.go:135] Filesystem UUIDs: map[24e67e82-2ff2-4606-b757-b53a7002ee9b:/dev/nvme0n1p4 383de8ec-6aa4-4a0d-b7bb-3c8065694dba:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 27 17:28:02.847592 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.847532 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 17:28:02.853157 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.853046 2564 manager.go:216] Machine: {Timestamp:2026-07-27 17:28:02.85119031 +0000 UTC m=+0.299880478 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097304 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b6725901c2008c44b9269d5f238c SystemUUID:ec22b672-5901-c200-8c44-b9269d5f238c BootID:378c2484-6bc3-4063-8541-f4ff93fb77a7 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:ee:a2:06:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:ee:a2:06:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:b5:20:58:b5:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 17:28:02.853157 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.853154 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 17:28:02.853283 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.853275 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 17:28:02.856343 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.856316 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 17:28:02.856471 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.856343 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 17:28:02.857176 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.857167 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 17:28:02.857205 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.857179 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 17:28:02.857205 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.857187 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 17:28:02.857846 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.857828 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 17:28:02.857927 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.857852 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 17:28:02.858916 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.858904 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 17:28:02.861597 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.861585 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 27 17:28:02.861709 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.861700 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 17:28:02.861748 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.861731 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 27 17:28:02.861748 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.861740 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 17:28:02.861810 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.861753 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 17:28:02.861810 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.861762 2564 kubelet.go:398] "Adding apiserver pod source" Jul 27 17:28:02.861810 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.861771 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 17:28:02.862127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.862113 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 17:28:02.862175 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.862138 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 17:28:02.864158 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.864139 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 17:28:02.866307 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.866292 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 17:28:02.866397 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.866337 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 17:28:02.868021 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868006 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868027 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868037 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868045 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868058 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868065 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868077 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868084 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 17:28:02.868093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868094 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 17:28:02.868369 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868103 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 17:28:02.868369 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868116 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 17:28:02.868369 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.868125 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 17:28:02.869559 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.869547 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 17:28:02.869612 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.869563 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 17:28:02.871973 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.871957 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 17:28:02.871973 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.871972 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 17:28:02.872104 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.872067 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 17:28:02.872840 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.872827 2564 server.go:1268] "Started kubelet" Jul 27 17:28:02.872941 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.872910 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 17:28:02.873007 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.872934 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 17:28:02.873054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.873043 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 17:28:02.873737 ip-10-0-131-71 systemd[1]: Started Kubernetes Kubelet. Jul 27 17:28:02.874791 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.874768 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 27 17:28:02.878135 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.878086 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 17:28:02.878773 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.878750 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 17:28:02.880344 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.880323 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qxr6c" Jul 27 17:28:02.880915 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.879972 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-71.ec2.internal.18c635a47c26aa19 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-71.ec2.internal,UID:ip-10-0-131-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-71.ec2.internal,},FirstTimestamp:2026-07-27 17:28:02.872797721 +0000 UTC m=+0.321487960,LastTimestamp:2026-07-27 17:28:02.872797721 +0000 UTC m=+0.321487960,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-71.ec2.internal,}" Jul 27 17:28:02.886533 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.886516 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 17:28:02.886533 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.886524 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 17:28:02.886655 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.886625 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:02.887174 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887153 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 17:28:02.887174 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887176 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 17:28:02.887324 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887160 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 17:28:02.887324 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887271 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 27 17:28:02.887324 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887278 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 27 17:28:02.887566 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887545 2564 factory.go:153] Registering CRI-O factory Jul 27 17:28:02.887566 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887560 2564 factory.go:223] Registration of the crio container factory successfully Jul 27 17:28:02.887670 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887606 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 17:28:02.887670 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887616 2564 factory.go:55] Registering systemd factory Jul 27 17:28:02.887670 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887625 2564 factory.go:223] Registration of the systemd container factory successfully Jul 27 17:28:02.887670 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887646 2564 factory.go:103] Registering Raw factory Jul 27 17:28:02.887670 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.887660 2564 manager.go:1196] Started watching for new ooms in manager Jul 27 17:28:02.888063 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.888046 2564 manager.go:349] Starting recovery of all containers Jul 27 17:28:02.888063 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.888055 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qxr6c" Jul 27 17:28:02.889200 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.889162 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 17:28:02.889307 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.889253 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 17:28:02.897104 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.897088 2564 manager.go:354] Recovery completed Jul 27 17:28:02.901170 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.901154 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:02.902946 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.902927 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:02.903026 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.902959 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:02.903026 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.902974 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:02.904008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.903993 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 17:28:02.904008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.904006 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 17:28:02.904105 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.904021 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 27 17:28:02.904105 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.903997 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:02.904763 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.904746 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:02.904844 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.904770 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:02.904844 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.904779 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:02.906579 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.906565 2564 policy_none.go:49] "None policy: Start" Jul 27 17:28:02.906579 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.906582 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 17:28:02.906661 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.906592 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 17:28:02.908487 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.908473 2564 policy_none.go:47] "Start" Jul 27 17:28:02.946895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.946721 2564 manager.go:337] "Starting Device Plugin manager" Jul 27 17:28:02.946965 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.946918 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 17:28:02.946965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.946931 2564 server.go:87] "Starting device plugin registration server" Jul 27 17:28:02.947173 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.947163 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 17:28:02.947207 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.947176 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 17:28:02.947317 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.947300 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 17:28:02.947422 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.947410 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 17:28:02.947474 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:02.947423 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 17:28:02.947910 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.947892 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 17:28:02.947975 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:02.947947 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:03.046600 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.046568 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 17:28:03.048537 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.047850 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 17:28:03.048537 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.047860 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.048537 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.047895 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 17:28:03.048537 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.047924 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 17:28:03.048537 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:03.047964 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 17:28:03.048779 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.048719 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.048779 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.048753 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.048779 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.048768 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.048914 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.048794 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-71.ec2.internal" Jul 27 17:28:03.050733 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.050714 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:03.058147 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.058132 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-71.ec2.internal" Jul 27 17:28:03.058251 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:03.058154 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-71.ec2.internal\": node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:03.092125 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:03.092104 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:03.148789 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.148766 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal"] Jul 27 17:28:03.148855 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.148840 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.149765 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.149750 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.149845 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.149785 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.149845 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.149802 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.151880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.151865 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.151994 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.151977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:03.152042 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.152013 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.152620 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.152605 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.152688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.152635 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.152688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.152646 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.152688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.152611 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.152823 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.152703 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.152823 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.152719 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.154777 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.154761 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 27 17:28:03.154860 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.154787 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.155411 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.155394 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.155490 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.155419 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.155490 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:03.155434 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.177837 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:03.177817 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-71.ec2.internal\" not found" node="ip-10-0-131-71.ec2.internal" Jul 27 17:28:03.182124 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:03.182109 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-71.ec2.internal\" not found" node="ip-10-0-131-71.ec2.internal" Jul 27 17:28:03.192926 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:03.192907 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:04.738210 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.738188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3f2e0ca0620d883c834a24cfc6c2200-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"f3f2e0ca0620d883c834a24cfc6c2200\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.738286 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.738214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f2e0ca0620d883c834a24cfc6c2200-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"f3f2e0ca0620d883c834a24cfc6c2200\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.738286 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.738230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3498390b3a0f07e184eb7f66c8538395-config\") pod \"kube-apiserver-proxy-ip-10-0-131-71.ec2.internal\" (UID: \"3498390b3a0f07e184eb7f66c8538395\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.743302 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:04.743276 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:04.839025 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.838941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3498390b3a0f07e184eb7f66c8538395-config\") pod \"kube-apiserver-proxy-ip-10-0-131-71.ec2.internal\" (UID: \"3498390b3a0f07e184eb7f66c8538395\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.839025 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.838989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3f2e0ca0620d883c834a24cfc6c2200-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"f3f2e0ca0620d883c834a24cfc6c2200\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.839025 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.839011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f2e0ca0620d883c834a24cfc6c2200-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"f3f2e0ca0620d883c834a24cfc6c2200\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.839208 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.839053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f2e0ca0620d883c834a24cfc6c2200-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"f3f2e0ca0620d883c834a24cfc6c2200\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.839208 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.839059 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3498390b3a0f07e184eb7f66c8538395-config\") pod \"kube-apiserver-proxy-ip-10-0-131-71.ec2.internal\" (UID: \"3498390b3a0f07e184eb7f66c8538395\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.839208 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.839059 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3f2e0ca0620d883c834a24cfc6c2200-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"f3f2e0ca0620d883c834a24cfc6c2200\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.844023 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:04.844007 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:04.931260 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.931225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.936244 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:04.936176 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 27 17:28:04.944902 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:04.944882 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.045491 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.045451 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.146027 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.145958 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.246520 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.246487 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.270592 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.270567 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 17:28:05.270974 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.270713 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 17:28:05.344965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.344918 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 17:23:04 +0000 UTC" deadline="2028-02-18 17:20:23.450738515 +0000 UTC" Jul 27 17:28:05.344965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.344958 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13703h52m18.105784155s" Jul 27 17:28:05.347064 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.347049 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.381984 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.381956 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:05.447715 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.447684 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.468954 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.468929 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:05.548791 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.548770 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.551913 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:05.551888 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f2e0ca0620d883c834a24cfc6c2200.slice/crio-ef5bcf9bdd19182bc7e4ea4c53c8ab5e28505f2fe4d44f7d4cb04931d4320120 WatchSource:0}: Error finding container ef5bcf9bdd19182bc7e4ea4c53c8ab5e28505f2fe4d44f7d4cb04931d4320120: Status 404 returned error can't find the container with id ef5bcf9bdd19182bc7e4ea4c53c8ab5e28505f2fe4d44f7d4cb04931d4320120 Jul 27 17:28:05.552558 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:05.552539 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3498390b3a0f07e184eb7f66c8538395.slice/crio-a4449df64b492ea8f39bf44c3805ac80130884d2067a0ebfd08e387f05f4e877 WatchSource:0}: Error finding container a4449df64b492ea8f39bf44c3805ac80130884d2067a0ebfd08e387f05f4e877: Status 404 returned error can't find the container with id a4449df64b492ea8f39bf44c3805ac80130884d2067a0ebfd08e387f05f4e877 Jul 27 17:28:05.556480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.556460 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:28:05.649802 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.649773 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.750392 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:05.750319 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 27 17:28:05.806267 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.806247 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:05.837549 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.837520 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 27 17:28:05.848820 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.848803 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 17:28:05.849597 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.849585 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 27 17:28:05.857940 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:05.857924 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 17:28:06.144234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.144170 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:06.312868 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.312838 2564 apiserver.go:52] "Watching apiserver" Jul 27 17:28:06.321966 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.321944 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 17:28:06.322356 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.322333 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-2cq5f","kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9cz8t","openshift-dns/node-resolver-dsrdc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal","openshift-multus/multus-gslm6","openshift-multus/network-metrics-daemon-6gcqw","openshift-network-diagnostics/network-check-target-w9rd7","kube-system/konnectivity-agent-4zv6d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc","openshift-image-registry/node-ca-xvbv6","openshift-multus/multus-additional-cni-plugins-xkvfz","openshift-network-operator/iptables-alerter-zndfc","openshift-ovn-kubernetes/ovnkube-node-r948w"] Jul 27 17:28:06.324796 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.324769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:06.324922 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.324864 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:06.327054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.327036 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.329302 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.329214 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.330104 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.330082 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 17:28:06.330251 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.330118 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v5lbk\"" Jul 27 17:28:06.330251 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.330208 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 17:28:06.330251 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.330239 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 17:28:06.331649 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.331633 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.331799 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.331778 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 17:28:06.331959 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.331935 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 17:28:06.332047 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.331967 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nqrvj\"" Jul 27 17:28:06.334108 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.334089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.334851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.334828 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27s85\"" Jul 27 17:28:06.334918 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.334858 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 17:28:06.335078 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.335064 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 17:28:06.336369 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.336353 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.337023 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.336647 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 17:28:06.337938 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.337922 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 17:28:06.338040 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.337936 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w7mcf\"" Jul 27 17:28:06.338100 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.338051 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 17:28:06.338216 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.338198 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 17:28:06.338290 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.338259 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 17:28:06.338445 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.338411 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 17:28:06.338669 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.338653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.339101 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.338741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 17:28:06.339213 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.339170 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 17:28:06.339274 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.339222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n4wh7\"" Jul 27 17:28:06.339331 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.339319 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 17:28:06.339331 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.339327 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 17:28:06.339845 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.339825 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 17:28:06.341222 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.340976 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.341436 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.341418 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 17:28:06.341775 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.341753 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 17:28:06.341872 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.341830 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-259qw\"" Jul 27 17:28:06.343287 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.343248 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 17:28:06.343522 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.343482 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 17:28:06.343995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.343817 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 17:28:06.343995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.343905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:06.343995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.343905 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mqmb8\"" Jul 27 17:28:06.343995 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.343967 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:06.346523 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.346504 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.348525 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-system-cni-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.348615 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-k8s-cni-cncf-io\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.348678 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-hosts-file\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.348678 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-var-lib-kubelet\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.348678 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-run-netns\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.348834 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-run-ovn-kubernetes\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.348834 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-ovnkube-config\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.348834 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be3196f-ed07-462b-a955-e78df488a631-serviceca\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.348834 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysconfig\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.348834 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysctl-conf\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348851 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-cnibin\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-os-release\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.348914 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-var-lib-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.348970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-tuned\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-daemon-config\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.349075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-etc-kubernetes\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8077354-83d5-412f-9b21-3b4878a000c5-agent-certs\") pod \"konnectivity-agent-4zv6d\" (UID: \"b8077354-83d5-412f-9b21-3b4878a000c5\") " pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349132 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-kubelet\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-node-log\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-sys\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349271 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cz8hx\"" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349372 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-cni-netd\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd97aaac-23af-44f8-865e-6cfeff89df73-ovn-node-metrics-cert\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-kubernetes\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.349480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-cni-multus\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-kubelet\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-netns\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-cni-bin\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-modprobe-d\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-cni-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-socket-dir-parent\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26g6q\" (UniqueName: \"kubernetes.io/projected/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-kube-api-access-26g6q\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349801 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-systemd-units\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-conf-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7mr\" (UniqueName: \"kubernetes.io/projected/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-kube-api-access-2m7mr\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.349994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-tmp-dir\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-env-overrides\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-ovnkube-script-lib\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-lib-modules\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-cni-bin\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-hostroot\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvj8\" (UniqueName: \"kubernetes.io/projected/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-kube-api-access-2gvj8\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvfz\" (UniqueName: \"kubernetes.io/projected/861d178f-ae27-4311-83a0-e0e3a9046297-kube-api-access-qsvfz\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-systemd\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-log-socket\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be3196f-ed07-462b-a955-e78df488a631-host\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-host\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rvh\" (UniqueName: \"kubernetes.io/projected/fd97aaac-23af-44f8-865e-6cfeff89df73-kube-api-access-47rvh\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.350819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mxk8\" (UniqueName: \"kubernetes.io/projected/4be3196f-ed07-462b-a955-e78df488a631-kube-api-access-6mxk8\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8077354-83d5-412f-9b21-3b4878a000c5-konnectivity-ca\") pod \"konnectivity-agent-4zv6d\" (UID: \"b8077354-83d5-412f-9b21-3b4878a000c5\") " pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-ovn\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350922 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-systemd\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.350955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-etc-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.351005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysctl-d\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.351134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-tmp\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.351219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-run\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.351251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-cni-binary-copy\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.351282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-multus-certs\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.351370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.351316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-slash\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.351803 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.351745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.354461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.354415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g9vfp\"" Jul 27 17:28:06.354461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.354437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 17:28:06.354781 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.354763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 17:28:06.373981 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.373961 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgsr6" Jul 27 17:28:06.379704 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.379684 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgsr6" Jul 27 17:28:06.437987 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.437966 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 17:28:06.451844 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-kubelet\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.451970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.451970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-host-slash\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.451970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-netns\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.451970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-cni-bin\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.451970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-modprobe-d\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.451970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-kubelet\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.451970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-cni-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.451984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-socket-dir-parent\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26g6q\" (UniqueName: \"kubernetes.io/projected/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-kube-api-access-26g6q\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-systemd-units\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-conf-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jpf\" (UniqueName: \"kubernetes.io/projected/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-kube-api-access-b7jpf\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7mr\" (UniqueName: \"kubernetes.io/projected/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-kube-api-access-2m7mr\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452231 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-netns\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-cni-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-cni-bin\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.452301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s72f\" (UniqueName: \"kubernetes.io/projected/74acacfd-72eb-4498-9437-385f6d5d5ec2-kube-api-access-6s72f\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-modprobe-d\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.452351 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-device-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.452421 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs podName:861d178f-ae27-4311-83a0-e0e3a9046297 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:06.9523911 +0000 UTC m=+2.951176900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs") pod "network-metrics-daemon-6gcqw" (UID: "861d178f-ae27-4311-83a0-e0e3a9046297") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-conf-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452475 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-socket-dir-parent\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-dbus\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-systemd-units\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-tmp-dir\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-env-overrides\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-ovnkube-script-lib\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-lib-modules\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-cni-bin\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-hostroot\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.452923 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvj8\" (UniqueName: \"kubernetes.io/projected/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-kube-api-access-2gvj8\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-cni-bin\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452782 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-hostroot\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-lib-modules\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-tmp-dir\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvfz\" (UniqueName: \"kubernetes.io/projected/861d178f-ae27-4311-83a0-e0e3a9046297-kube-api-access-qsvfz\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-systemd\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-log-socket\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be3196f-ed07-462b-a955-e78df488a631-host\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452968 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-systemd\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.452984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-host\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-iptables-alerter-script\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-log-socket\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5chh\" (UniqueName: \"kubernetes.io/projected/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-kube-api-access-d5chh\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rvh\" (UniqueName: \"kubernetes.io/projected/fd97aaac-23af-44f8-865e-6cfeff89df73-kube-api-access-47rvh\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be3196f-ed07-462b-a955-e78df488a631-host\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.453646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-host\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mxk8\" (UniqueName: \"kubernetes.io/projected/4be3196f-ed07-462b-a955-e78df488a631-kube-api-access-6mxk8\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8077354-83d5-412f-9b21-3b4878a000c5-konnectivity-ca\") pod \"konnectivity-agent-4zv6d\" (UID: \"b8077354-83d5-412f-9b21-3b4878a000c5\") " pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-system-cni-dir\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453180 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-env-overrides\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-cnibin\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-ovnkube-script-lib\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-os-release\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-registration-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-kubelet-config\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-ovn\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453469 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-run-ovn\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-systemd\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453543 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-systemd\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-sys-fs\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-etc-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.454358 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysctl-d\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8077354-83d5-412f-9b21-3b4878a000c5-konnectivity-ca\") pod \"konnectivity-agent-4zv6d\" (UID: \"b8077354-83d5-412f-9b21-3b4878a000c5\") " pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-tmp\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-etc-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-run\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysctl-d\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-cni-binary-copy\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-run\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453810 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-multus-certs\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-socket-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-slash\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453876 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-multus-certs\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-system-cni-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-slash\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.453991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-k8s-cni-cncf-io\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-system-cni-dir\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-hosts-file\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-var-lib-kubelet\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-hosts-file\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-run-netns\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-run-k8s-cni-cncf-io\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-var-lib-kubelet\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-run-ovn-kubernetes\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-run-netns\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-ovnkube-config\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be3196f-ed07-462b-a955-e78df488a631-serviceca\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysconfig\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-cni-binary-copy\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-run-ovn-kubernetes\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysctl-conf\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysconfig\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-cnibin\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-os-release\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-var-lib-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.455770 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454485 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-sysctl-conf\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-cnibin\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-tuned\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-daemon-config\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-os-release\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-etc-kubernetes\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8077354-83d5-412f-9b21-3b4878a000c5-agent-certs\") pod \"konnectivity-agent-4zv6d\" (UID: \"b8077354-83d5-412f-9b21-3b4878a000c5\") " pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be3196f-ed07-462b-a955-e78df488a631-serviceca\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-var-lib-openvswitch\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-kubelet\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-node-log\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-sys\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-node-log\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-etc-kubernetes\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-cni-netd\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-sys\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.456428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd97aaac-23af-44f8-865e-6cfeff89df73-ovnkube-config\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-cni-netd\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd97aaac-23af-44f8-865e-6cfeff89df73-host-kubelet\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd97aaac-23af-44f8-865e-6cfeff89df73-ovn-node-metrics-cert\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-kubernetes\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-cni-multus\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-kubernetes\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.454975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-host-var-lib-cni-multus\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.457342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.455016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-multus-daemon-config\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.457701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.457663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-tmp\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.457759 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.457699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-etc-tuned\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.457809 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.457788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8077354-83d5-412f-9b21-3b4878a000c5-agent-certs\") pod \"konnectivity-agent-4zv6d\" (UID: \"b8077354-83d5-412f-9b21-3b4878a000c5\") " pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.457983 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.457937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd97aaac-23af-44f8-865e-6cfeff89df73-ovn-node-metrics-cert\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.463796 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.463774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26g6q\" (UniqueName: \"kubernetes.io/projected/c3ecd635-1afa-47cb-9dd1-e43b956e43a6-kube-api-access-26g6q\") pod \"multus-gslm6\" (UID: \"c3ecd635-1afa-47cb-9dd1-e43b956e43a6\") " pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.465798 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.465773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7mr\" (UniqueName: \"kubernetes.io/projected/6d3767a9-6ba7-481d-b6bf-b2f76eb5a679-kube-api-access-2m7mr\") pod \"node-resolver-dsrdc\" (UID: \"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679\") " pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.466170 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.466130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvfz\" (UniqueName: \"kubernetes.io/projected/861d178f-ae27-4311-83a0-e0e3a9046297-kube-api-access-qsvfz\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:06.466462 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.466435 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvj8\" (UniqueName: \"kubernetes.io/projected/3d70e7cf-9907-43e7-96e9-b7b421bbaffa-kube-api-access-2gvj8\") pod \"tuned-9cz8t\" (UID: \"3d70e7cf-9907-43e7-96e9-b7b421bbaffa\") " pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.466924 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.466903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mxk8\" (UniqueName: \"kubernetes.io/projected/4be3196f-ed07-462b-a955-e78df488a631-kube-api-access-6mxk8\") pod \"node-ca-xvbv6\" (UID: \"4be3196f-ed07-462b-a955-e78df488a631\") " pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.467271 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.467252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rvh\" (UniqueName: \"kubernetes.io/projected/fd97aaac-23af-44f8-865e-6cfeff89df73-kube-api-access-47rvh\") pod \"ovnkube-node-r948w\" (UID: \"fd97aaac-23af-44f8-865e-6cfeff89df73\") " pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.502817 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.502778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" event={"ID":"f3f2e0ca0620d883c834a24cfc6c2200","Type":"ContainerStarted","Data":"ef5bcf9bdd19182bc7e4ea4c53c8ab5e28505f2fe4d44f7d4cb04931d4320120"} Jul 27 17:28:06.506751 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.504241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" event={"ID":"3498390b3a0f07e184eb7f66c8538395","Type":"ContainerStarted","Data":"a4449df64b492ea8f39bf44c3805ac80130884d2067a0ebfd08e387f05f4e877"} Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s72f\" (UniqueName: \"kubernetes.io/projected/74acacfd-72eb-4498-9437-385f6d5d5ec2-kube-api-access-6s72f\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-device-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-dbus\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-iptables-alerter-script\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5chh\" (UniqueName: \"kubernetes.io/projected/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-kube-api-access-d5chh\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-system-cni-dir\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.555986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-cnibin\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-os-release\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-registration-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-kubelet-config\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-sys-fs\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-socket-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.556415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-host-slash\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jpf\" (UniqueName: \"kubernetes.io/projected/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-kube-api-access-b7jpf\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.556564 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.556648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret podName:b53269e6-9bf1-4b56-8c80-73a3e3f630dd nodeName:}" failed. No retries permitted until 2026-07-27 17:28:07.056629477 +0000 UTC m=+3.055415276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret") pod "global-pull-secret-syncer-2cq5f" (UID: "b53269e6-9bf1-4b56-8c80-73a3e3f630dd") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-kubelet-config\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556847 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.556980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-cnibin\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557075 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-os-release\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-dbus\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:06.557318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74acacfd-72eb-4498-9437-385f6d5d5ec2-system-cni-dir\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-sys-fs\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557288 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-registration-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-device-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-host-slash\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-socket-dir\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-iptables-alerter-script\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.558165 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.557908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.558579 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.558267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74acacfd-72eb-4498-9437-385f6d5d5ec2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.562991 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.562971 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:06.562991 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.562992 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:06.563119 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.563001 2564 projected.go:196] Error preparing data for projected volume kube-api-access-c2wfv for pod openshift-network-diagnostics/network-check-target-w9rd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:06.563119 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.563050 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv podName:736e470f-079a-4aea-b75c-c4ac3cf684bf nodeName:}" failed. No retries permitted until 2026-07-27 17:28:07.063036568 +0000 UTC m=+3.061822372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c2wfv" (UniqueName: "kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv") pod "network-check-target-w9rd7" (UID: "736e470f-079a-4aea-b75c-c4ac3cf684bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:06.565290 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.565266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s72f\" (UniqueName: \"kubernetes.io/projected/74acacfd-72eb-4498-9437-385f6d5d5ec2-kube-api-access-6s72f\") pod \"multus-additional-cni-plugins-xkvfz\" (UID: \"74acacfd-72eb-4498-9437-385f6d5d5ec2\") " pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.565611 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.565593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5chh\" (UniqueName: \"kubernetes.io/projected/027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd-kube-api-access-d5chh\") pod \"iptables-alerter-zndfc\" (UID: \"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd\") " pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.565931 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.565910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jpf\" (UniqueName: \"kubernetes.io/projected/8fd0d5fc-fc4e-4422-93f9-e147bfedd116-kube-api-access-b7jpf\") pod \"aws-ebs-csi-driver-node-8zqfc\" (UID: \"8fd0d5fc-fc4e-4422-93f9-e147bfedd116\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.637931 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.637899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvbv6" Jul 27 17:28:06.646319 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.646297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" Jul 27 17:28:06.657906 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.657884 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dsrdc" Jul 27 17:28:06.661525 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.661505 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:06.669231 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.669209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gslm6" Jul 27 17:28:06.676807 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.676787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:06.684298 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.684277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zndfc" Jul 27 17:28:06.695805 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.695786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" Jul 27 17:28:06.701225 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.701204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" Jul 27 17:28:06.913264 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.913225 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8077354_83d5_412f_9b21_3b4878a000c5.slice/crio-24a50b18c218d18ee8cdbddbf2b0e93a1823bdb87b462cbad8e1615dbe0d69f1 WatchSource:0}: Error finding container 24a50b18c218d18ee8cdbddbf2b0e93a1823bdb87b462cbad8e1615dbe0d69f1: Status 404 returned error can't find the container with id 24a50b18c218d18ee8cdbddbf2b0e93a1823bdb87b462cbad8e1615dbe0d69f1 Jul 27 17:28:06.914627 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.914603 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd97aaac_23af_44f8_865e_6cfeff89df73.slice/crio-436e6f94d888a993f0331abd0cd132413c7c13f51638aa28bfbc3d89cb9548d4 WatchSource:0}: Error finding container 436e6f94d888a993f0331abd0cd132413c7c13f51638aa28bfbc3d89cb9548d4: Status 404 returned error can't find the container with id 436e6f94d888a993f0331abd0cd132413c7c13f51638aa28bfbc3d89cb9548d4 Jul 27 17:28:06.920654 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.920616 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be3196f_ed07_462b_a955_e78df488a631.slice/crio-f05728060db8127af7b129cd1612b1ae9de3a6e25cab2f26015fafc42488c42b WatchSource:0}: Error finding container f05728060db8127af7b129cd1612b1ae9de3a6e25cab2f26015fafc42488c42b: Status 404 returned error can't find the container with id f05728060db8127af7b129cd1612b1ae9de3a6e25cab2f26015fafc42488c42b Jul 27 17:28:06.923800 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.923778 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d70e7cf_9907_43e7_96e9_b7b421bbaffa.slice/crio-f2362435079bb8154ad1bf5adf23afc2c2b47961cbf209be9987eb0862b82184 WatchSource:0}: Error finding container f2362435079bb8154ad1bf5adf23afc2c2b47961cbf209be9987eb0862b82184: Status 404 returned error can't find the container with id f2362435079bb8154ad1bf5adf23afc2c2b47961cbf209be9987eb0862b82184 Jul 27 17:28:06.924509 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.924483 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74acacfd_72eb_4498_9437_385f6d5d5ec2.slice/crio-121ac8e8f4658ecdd8108e7f5146a9158e88958f9072070b9d1be39f898529b8 WatchSource:0}: Error finding container 121ac8e8f4658ecdd8108e7f5146a9158e88958f9072070b9d1be39f898529b8: Status 404 returned error can't find the container with id 121ac8e8f4658ecdd8108e7f5146a9158e88958f9072070b9d1be39f898529b8 Jul 27 17:28:06.925528 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.925504 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd0d5fc_fc4e_4422_93f9_e147bfedd116.slice/crio-5cde7c36bf39f738ed660c8b800f47f72ce877dd7fd4509585b268ac0d05d27b WatchSource:0}: Error finding container 5cde7c36bf39f738ed660c8b800f47f72ce877dd7fd4509585b268ac0d05d27b: Status 404 returned error can't find the container with id 5cde7c36bf39f738ed660c8b800f47f72ce877dd7fd4509585b268ac0d05d27b Jul 27 17:28:06.926317 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.926278 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ecd635_1afa_47cb_9dd1_e43b956e43a6.slice/crio-97f541d5e5587537618c2f9a5cb9757c035d7e278978d1bbe2369daf4caa5608 WatchSource:0}: Error finding container 97f541d5e5587537618c2f9a5cb9757c035d7e278978d1bbe2369daf4caa5608: Status 404 returned error can't find the container with id 97f541d5e5587537618c2f9a5cb9757c035d7e278978d1bbe2369daf4caa5608 Jul 27 17:28:06.927477 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.927439 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027bfb55_a6e1_4a7c_ba3a_59e4adaa97fd.slice/crio-c09df379124538d241b216d8f09a5152247e67bd8f26d9b3539b15acf813a53e WatchSource:0}: Error finding container c09df379124538d241b216d8f09a5152247e67bd8f26d9b3539b15acf813a53e: Status 404 returned error can't find the container with id c09df379124538d241b216d8f09a5152247e67bd8f26d9b3539b15acf813a53e Jul 27 17:28:06.928931 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:06.928430 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3767a9_6ba7_481d_b6bf_b2f76eb5a679.slice/crio-bc67e3abaad5240df54fbe9ca5fdd0abae93a0cb82ab17a2ded2b56c4557c838 WatchSource:0}: Error finding container bc67e3abaad5240df54fbe9ca5fdd0abae93a0cb82ab17a2ded2b56c4557c838: Status 404 returned error can't find the container with id bc67e3abaad5240df54fbe9ca5fdd0abae93a0cb82ab17a2ded2b56c4557c838 Jul 27 17:28:06.959857 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:06.959826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:06.959948 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.959932 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:06.960009 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:06.959993 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs podName:861d178f-ae27-4311-83a0-e0e3a9046297 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:07.959973803 +0000 UTC m=+3.958759607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs") pod "network-metrics-daemon-6gcqw" (UID: "861d178f-ae27-4311-83a0-e0e3a9046297") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:07.061102 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.061062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:07.061270 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.061204 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:07.061330 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.061281 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret podName:b53269e6-9bf1-4b56-8c80-73a3e3f630dd nodeName:}" failed. No retries permitted until 2026-07-27 17:28:08.061261343 +0000 UTC m=+4.060047161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret") pod "global-pull-secret-syncer-2cq5f" (UID: "b53269e6-9bf1-4b56-8c80-73a3e3f630dd") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:07.162258 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.162186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:07.162410 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.162332 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:07.162410 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.162356 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:07.162410 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.162369 2564 projected.go:196] Error preparing data for projected volume kube-api-access-c2wfv for pod openshift-network-diagnostics/network-check-target-w9rd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:07.162573 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.162435 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv podName:736e470f-079a-4aea-b75c-c4ac3cf684bf nodeName:}" failed. No retries permitted until 2026-07-27 17:28:08.162414493 +0000 UTC m=+4.161200314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2wfv" (UniqueName: "kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv") pod "network-check-target-w9rd7" (UID: "736e470f-079a-4aea-b75c-c4ac3cf684bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:07.383121 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.383079 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:06 +0000 UTC" deadline="2027-12-29 19:16:46.845506614 +0000 UTC" Jul 27 17:28:07.383121 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.383119 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12481h48m39.462394907s" Jul 27 17:28:07.498438 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.498355 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:07.498607 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.498510 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:07.498959 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.498818 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:07.498959 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.498911 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:07.508720 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.508670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dsrdc" event={"ID":"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679","Type":"ContainerStarted","Data":"bc67e3abaad5240df54fbe9ca5fdd0abae93a0cb82ab17a2ded2b56c4557c838"} Jul 27 17:28:07.510326 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.510292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" event={"ID":"8fd0d5fc-fc4e-4422-93f9-e147bfedd116","Type":"ContainerStarted","Data":"5cde7c36bf39f738ed660c8b800f47f72ce877dd7fd4509585b268ac0d05d27b"} Jul 27 17:28:07.511849 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.511798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" event={"ID":"3d70e7cf-9907-43e7-96e9-b7b421bbaffa","Type":"ContainerStarted","Data":"f2362435079bb8154ad1bf5adf23afc2c2b47961cbf209be9987eb0862b82184"} Jul 27 17:28:07.513361 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.513305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvbv6" event={"ID":"4be3196f-ed07-462b-a955-e78df488a631","Type":"ContainerStarted","Data":"f05728060db8127af7b129cd1612b1ae9de3a6e25cab2f26015fafc42488c42b"} Jul 27 17:28:07.515048 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.515022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zv6d" event={"ID":"b8077354-83d5-412f-9b21-3b4878a000c5","Type":"ContainerStarted","Data":"24a50b18c218d18ee8cdbddbf2b0e93a1823bdb87b462cbad8e1615dbe0d69f1"} Jul 27 17:28:07.516660 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.516634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zndfc" event={"ID":"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd","Type":"ContainerStarted","Data":"c09df379124538d241b216d8f09a5152247e67bd8f26d9b3539b15acf813a53e"} Jul 27 17:28:07.518304 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.518285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gslm6" event={"ID":"c3ecd635-1afa-47cb-9dd1-e43b956e43a6","Type":"ContainerStarted","Data":"97f541d5e5587537618c2f9a5cb9757c035d7e278978d1bbe2369daf4caa5608"} Jul 27 17:28:07.519555 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.519533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerStarted","Data":"121ac8e8f4658ecdd8108e7f5146a9158e88958f9072070b9d1be39f898529b8"} Jul 27 17:28:07.521070 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.521042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"436e6f94d888a993f0331abd0cd132413c7c13f51638aa28bfbc3d89cb9548d4"} Jul 27 17:28:07.969159 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:07.968421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:07.969159 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.968762 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:07.969159 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:07.968820 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs podName:861d178f-ae27-4311-83a0-e0e3a9046297 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:09.968802246 +0000 UTC m=+5.967588064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs") pod "network-metrics-daemon-6gcqw" (UID: "861d178f-ae27-4311-83a0-e0e3a9046297") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:08.069952 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.069322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:08.069952 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:08.069525 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:08.069952 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:08.069592 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret podName:b53269e6-9bf1-4b56-8c80-73a3e3f630dd nodeName:}" failed. No retries permitted until 2026-07-27 17:28:10.069572352 +0000 UTC m=+6.068358158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret") pod "global-pull-secret-syncer-2cq5f" (UID: "b53269e6-9bf1-4b56-8c80-73a3e3f630dd") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:08.170206 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.170115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:08.170359 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:08.170305 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:08.170359 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:08.170329 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:08.170359 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:08.170342 2564 projected.go:196] Error preparing data for projected volume kube-api-access-c2wfv for pod openshift-network-diagnostics/network-check-target-w9rd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:08.170488 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:08.170406 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv podName:736e470f-079a-4aea-b75c-c4ac3cf684bf nodeName:}" failed. No retries permitted until 2026-07-27 17:28:10.170387479 +0000 UTC m=+6.169173292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2wfv" (UniqueName: "kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv") pod "network-check-target-w9rd7" (UID: "736e470f-079a-4aea-b75c-c4ac3cf684bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:08.384220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.384132 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:06 +0000 UTC" deadline="2028-02-02 11:28:45.393121409 +0000 UTC" Jul 27 17:28:08.384220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.384191 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13314h0m37.008935651s" Jul 27 17:28:08.499007 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.498971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:08.499184 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:08.499114 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:08.531663 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.531579 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3f2e0ca0620d883c834a24cfc6c2200" containerID="e87fce328fea409d3374145aacc17014fa6b6c18522abe0afb7a27dd2e4205d9" exitCode=0 Jul 27 17:28:08.531840 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.531695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" event={"ID":"f3f2e0ca0620d883c834a24cfc6c2200","Type":"ContainerDied","Data":"e87fce328fea409d3374145aacc17014fa6b6c18522abe0afb7a27dd2e4205d9"} Jul 27 17:28:08.536055 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:08.536020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" event={"ID":"3498390b3a0f07e184eb7f66c8538395","Type":"ContainerStarted","Data":"636db14c068303399048df03ea1d61e78f0b7eebf206c406a81114b61b8bf0b8"} Jul 27 17:28:09.498092 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:09.498063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:09.498511 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:09.498217 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:09.498631 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:09.498610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:09.498742 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:09.498720 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:09.987962 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:09.987890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:09.988087 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:09.988036 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:09.988121 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:09.988092 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs podName:861d178f-ae27-4311-83a0-e0e3a9046297 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:13.988074785 +0000 UTC m=+9.986860600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs") pod "network-metrics-daemon-6gcqw" (UID: "861d178f-ae27-4311-83a0-e0e3a9046297") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:10.089239 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:10.089202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:10.089427 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:10.089367 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:10.089519 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:10.089428 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret podName:b53269e6-9bf1-4b56-8c80-73a3e3f630dd nodeName:}" failed. No retries permitted until 2026-07-27 17:28:14.089410863 +0000 UTC m=+10.088196667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret") pod "global-pull-secret-syncer-2cq5f" (UID: "b53269e6-9bf1-4b56-8c80-73a3e3f630dd") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:10.190268 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:10.189649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:10.190268 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:10.189796 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:10.190268 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:10.189816 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:10.190268 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:10.189828 2564 projected.go:196] Error preparing data for projected volume kube-api-access-c2wfv for pod openshift-network-diagnostics/network-check-target-w9rd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:10.190268 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:10.189908 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv podName:736e470f-079a-4aea-b75c-c4ac3cf684bf nodeName:}" failed. No retries permitted until 2026-07-27 17:28:14.189862343 +0000 UTC m=+10.188648144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2wfv" (UniqueName: "kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv") pod "network-check-target-w9rd7" (UID: "736e470f-079a-4aea-b75c-c4ac3cf684bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:10.501112 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:10.501085 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:10.501569 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:10.501223 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:10.550038 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:10.549408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" event={"ID":"f3f2e0ca0620d883c834a24cfc6c2200","Type":"ContainerStarted","Data":"6c9a4e82a58810ed6383bb136afe909ebda2ff871a918df113e194fd3d273109"} Jul 27 17:28:10.552072 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:10.551944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zv6d" event={"ID":"b8077354-83d5-412f-9b21-3b4878a000c5","Type":"ContainerStarted","Data":"94f4bcea728aa5d4254fa1a137e669e62f80e741c0ab573f8cebda1a229467ff"} Jul 27 17:28:10.565604 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:10.565437 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" podStartSLOduration=5.565422379 podStartE2EDuration="5.565422379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:08.575224456 +0000 UTC m=+4.574010290" watchObservedRunningTime="2026-07-27 17:28:10.565422379 +0000 UTC m=+6.564208230" Jul 27 17:28:10.566153 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:10.566050 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" podStartSLOduration=5.566037702 podStartE2EDuration="5.566037702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:10.565906924 +0000 UTC m=+6.564692746" watchObservedRunningTime="2026-07-27 17:28:10.566037702 +0000 UTC m=+6.564823525" Jul 27 17:28:11.498915 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:11.498374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:11.498915 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:11.498485 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:11.498915 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:11.498794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:11.498915 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:11.498872 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:12.498425 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:12.498391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:12.498843 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:12.498528 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:13.498539 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:13.498503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:13.499003 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:13.498562 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:13.499003 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:13.498661 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:13.499124 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:13.499081 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:14.025792 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:14.025375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:14.025792 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.025537 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:14.025792 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.025604 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs podName:861d178f-ae27-4311-83a0-e0e3a9046297 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:22.025584597 +0000 UTC m=+18.024370413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs") pod "network-metrics-daemon-6gcqw" (UID: "861d178f-ae27-4311-83a0-e0e3a9046297") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:14.126206 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:14.126166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:14.126394 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.126369 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:14.126470 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.126453 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret podName:b53269e6-9bf1-4b56-8c80-73a3e3f630dd nodeName:}" failed. No retries permitted until 2026-07-27 17:28:22.126431938 +0000 UTC m=+18.125217755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret") pod "global-pull-secret-syncer-2cq5f" (UID: "b53269e6-9bf1-4b56-8c80-73a3e3f630dd") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:14.226706 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:14.226669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:14.226875 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.226811 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:14.226875 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.226830 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:14.226875 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.226839 2564 projected.go:196] Error preparing data for projected volume kube-api-access-c2wfv for pod openshift-network-diagnostics/network-check-target-w9rd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:14.227076 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.226888 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv podName:736e470f-079a-4aea-b75c-c4ac3cf684bf nodeName:}" failed. No retries permitted until 2026-07-27 17:28:22.226869887 +0000 UTC m=+18.225655687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2wfv" (UniqueName: "kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv") pod "network-check-target-w9rd7" (UID: "736e470f-079a-4aea-b75c-c4ac3cf684bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:14.500980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:14.500907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:14.501479 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:14.501042 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:15.193878 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:15.193663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:15.194381 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:15.194335 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:15.211460 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:15.210937 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4zv6d" podStartSLOduration=8.68489652 podStartE2EDuration="11.210919374s" totalImagesPullingTime="2.526022854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:10.583489441 +0000 UTC m=+6.582275284" watchObservedRunningTime="2026-07-27 17:28:15.210919374 +0000 UTC m=+11.209705197" Jul 27 17:28:15.499166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:15.499017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:15.499321 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:15.499159 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:15.499321 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:15.499178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:15.499430 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:15.499336 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:16.498970 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:16.498936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:16.499475 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:16.499065 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:17.499059 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:17.499019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:17.499426 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:17.499019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:17.499426 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:17.499188 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:17.499426 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:17.499216 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:18.376872 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:18.376840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:18.443494 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:18.443454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:18.498516 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:18.498482 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:18.498711 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:18.498633 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:19.498958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:19.498924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:19.499388 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:19.498925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:19.499388 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:19.499046 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:19.499388 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:19.499170 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:20.461240 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:20.461207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:20.498248 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:20.498213 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:20.498427 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:20.498341 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:20.561721 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:20.561686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:21.498449 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:21.498418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:21.498624 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:21.498521 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:21.498624 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:21.498574 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:21.498693 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:21.498656 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:22.082503 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:22.082467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:22.082919 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.082616 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:22.082919 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.082691 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs podName:861d178f-ae27-4311-83a0-e0e3a9046297 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:38.082673091 +0000 UTC m=+34.081458908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs") pod "network-metrics-daemon-6gcqw" (UID: "861d178f-ae27-4311-83a0-e0e3a9046297") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:22.090958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:22.090917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:22.183350 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:22.183311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:22.183510 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.183438 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:22.183510 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.183509 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret podName:b53269e6-9bf1-4b56-8c80-73a3e3f630dd nodeName:}" failed. No retries permitted until 2026-07-27 17:28:38.183493832 +0000 UTC m=+34.182279631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret") pod "global-pull-secret-syncer-2cq5f" (UID: "b53269e6-9bf1-4b56-8c80-73a3e3f630dd") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:22.192276 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:22.192247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:22.284578 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:22.284544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:22.284772 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.284699 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:22.284772 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.284716 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:22.284772 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.284725 2564 projected.go:196] Error preparing data for projected volume kube-api-access-c2wfv for pod openshift-network-diagnostics/network-check-target-w9rd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:22.284943 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.284786 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv podName:736e470f-079a-4aea-b75c-c4ac3cf684bf nodeName:}" failed. No retries permitted until 2026-07-27 17:28:38.284767127 +0000 UTC m=+34.283552928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2wfv" (UniqueName: "kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv") pod "network-check-target-w9rd7" (UID: "736e470f-079a-4aea-b75c-c4ac3cf684bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:22.498784 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:22.498753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:22.498944 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:22.498868 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:22.810803 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:22.810724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:23.499034 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:23.499000 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:23.499034 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:23.499036 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:23.499552 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:23.499132 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:23.499552 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:23.499275 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:24.499297 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:24.499262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:24.499759 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:24.499377 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:25.500186 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:25.499401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:25.500186 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:25.499540 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:25.500186 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:25.499398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:25.500186 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:25.499648 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:25.586849 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:25.586044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" event={"ID":"3d70e7cf-9907-43e7-96e9-b7b421bbaffa","Type":"ContainerStarted","Data":"e4a7c75d580a36122bfb39c5e841bd7f21cb1389d25489503684ad2d145100b6"} Jul 27 17:28:25.608386 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:25.607579 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9cz8t" podStartSLOduration=3.072289907 podStartE2EDuration="21.607557333s" totalImagesPullingTime="18.535267426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:25.607274704 +0000 UTC m=+21.606060525" watchObservedRunningTime="2026-07-27 17:28:25.607557333 +0000 UTC m=+21.606343155" Jul 27 17:28:26.498742 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.498717 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:26.498883 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:26.498816 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:26.589101 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.589065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dsrdc" event={"ID":"6d3767a9-6ba7-481d-b6bf-b2f76eb5a679","Type":"ContainerStarted","Data":"3805645d10cd4784ae558db7e48e17f4265b5336e325fe533f8928c166f020d1"} Jul 27 17:28:26.590576 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.590552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" event={"ID":"8fd0d5fc-fc4e-4422-93f9-e147bfedd116","Type":"ContainerStarted","Data":"9ead71ab2c8fc0f0fb4f155727e8a88bda1368f50f364d1a3a3a23eae0aaa7ef"} Jul 27 17:28:26.591918 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.591897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvbv6" event={"ID":"4be3196f-ed07-462b-a955-e78df488a631","Type":"ContainerStarted","Data":"69046e255d674ca550348d091caf8006f4e45bda2cc1469bf799593f8d918f92"} Jul 27 17:28:26.593384 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.593359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gslm6" event={"ID":"c3ecd635-1afa-47cb-9dd1-e43b956e43a6","Type":"ContainerStarted","Data":"522ddc80e1eee03f5550e334bb7f3fbcd7bab56a56381dd97ce1038b96091dfc"} Jul 27 17:28:26.594826 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.594806 2564 generic.go:356] "Generic (PLEG): container finished" podID="74acacfd-72eb-4498-9437-385f6d5d5ec2" containerID="ba65ab3a2d71ceab62ccd6ea6c00455a505b4068a424cee0869064da60de8d0b" exitCode=0 Jul 27 17:28:26.594901 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.594883 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerDied","Data":"ba65ab3a2d71ceab62ccd6ea6c00455a505b4068a424cee0869064da60de8d0b"} Jul 27 17:28:26.599597 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.599566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"a8306b43fce599579bcc71434701f598ca5888666f0912403313abd099023852"} Jul 27 17:28:26.599694 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.599647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"c8716ab340af36a80df576400bebb909de0aa5ee7ca05101d145956202c2c1da"} Jul 27 17:28:26.599755 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.599711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"d306a76469dac8d41af00b605bddf6087a25884fdf16d3f69bce2f01580f0acd"} Jul 27 17:28:26.599810 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.599764 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"af8226bc5fe1720ec30e90981b5a3e158d4de4ff218a6f1dcb27e737a1724617"} Jul 27 17:28:26.599867 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.599815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"55156ad47dd54b7fdb5afeba82726b2c16aa5dc5ba300e2cc1a4c9a9c8111893"} Jul 27 17:28:26.599867 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.599862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"be7215f6a1262d2f1224240047c8da3bff76edbf1d149e0639b13546363c47e4"} Jul 27 17:28:26.605846 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.605808 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dsrdc" podStartSLOduration=4.070834951 podStartE2EDuration="22.605796708s" totalImagesPullingTime="18.534961757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:26.605098856 +0000 UTC m=+22.603884685" watchObservedRunningTime="2026-07-27 17:28:26.605796708 +0000 UTC m=+22.604582529" Jul 27 17:28:26.620310 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.620266 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xvbv6" podStartSLOduration=4.076015381 podStartE2EDuration="22.620253533s" totalImagesPullingTime="18.544238152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:26.619903284 +0000 UTC m=+22.618689102" watchObservedRunningTime="2026-07-27 17:28:26.620253533 +0000 UTC m=+22.619039354" Jul 27 17:28:26.640838 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.640788 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gslm6" podStartSLOduration=4.046780391 podStartE2EDuration="22.640771678s" totalImagesPullingTime="18.593991287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:26.640723386 +0000 UTC m=+22.639509208" watchObservedRunningTime="2026-07-27 17:28:26.640771678 +0000 UTC m=+22.639557499" Jul 27 17:28:26.763401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.763321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:26.824401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:26.824368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:27.499080 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:27.499050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:27.499080 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:27.499088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:27.499557 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:27.499191 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:27.499634 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:27.499605 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:27.604290 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:27.604245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zndfc" event={"ID":"027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd","Type":"ContainerStarted","Data":"49369f7c8716ee1ffe1d3231a7303d64bbb1df036e8a0d14bf3efdd5dd6ca6f9"} Jul 27 17:28:27.620341 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:27.620291 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zndfc" podStartSLOduration=5.077336058 podStartE2EDuration="23.620272739s" totalImagesPullingTime="18.542936681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:27.619955853 +0000 UTC m=+23.618741696" watchObservedRunningTime="2026-07-27 17:28:27.620272739 +0000 UTC m=+23.619058562" Jul 27 17:28:27.739368 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:27.739333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:27.798884 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:27.798763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:28.255643 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:28.255609 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 17:28:28.411095 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:28.410953 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:28:28.255642453Z","UUID":"8169ab40-6c23-4732-9d13-7477a672a50a","Handler":null,"Name":"","Endpoint":""} Jul 27 17:28:28.412535 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:28.412514 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 17:28:28.412648 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:28.412551 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 17:28:28.498235 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:28.498195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:28.498383 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:28.498330 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:28.608467 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:28.608254 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" event={"ID":"8fd0d5fc-fc4e-4422-93f9-e147bfedd116","Type":"ContainerStarted","Data":"1c6862a7fa1ac5a41aa47f18b88cb88a069330f651e8b6906df6da54377c0925"} Jul 27 17:28:28.612530 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:28.612497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"7de50403cbe4073dc48c4ffb8db4817634b8f013a3bec858f083aa14475f5112"} Jul 27 17:28:29.498639 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:29.498604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:29.498823 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:29.498621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:29.498823 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:29.498749 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:29.498945 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:29.498818 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:30.498740 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:30.498552 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:30.499115 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:30.498861 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:31.498606 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.498428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:31.498734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.498459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:31.498734 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:31.498692 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:31.499346 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:31.498755 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:31.621283 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.621209 2564 generic.go:356] "Generic (PLEG): container finished" podID="74acacfd-72eb-4498-9437-385f6d5d5ec2" containerID="e2e186cef78242e5b5a400e705749e5e868bffaf4d11f7742c8db41ef075ab6e" exitCode=0 Jul 27 17:28:31.621405 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.621307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerDied","Data":"e2e186cef78242e5b5a400e705749e5e868bffaf4d11f7742c8db41ef075ab6e"} Jul 27 17:28:31.627885 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.627860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" event={"ID":"fd97aaac-23af-44f8-865e-6cfeff89df73","Type":"ContainerStarted","Data":"2644f16c7e1a03348f711fda459d07c800fb21cd09ff70e2f03da52f8b6af98a"} Jul 27 17:28:31.628053 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.628034 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:31.628132 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.628104 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:31.628204 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.628169 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:31.630201 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.630181 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" event={"ID":"8fd0d5fc-fc4e-4422-93f9-e147bfedd116","Type":"ContainerStarted","Data":"300213f5a0b2eda4f88af340e0ab97dad114de831d62d65d0b577b2e07ac8041"} Jul 27 17:28:31.643401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.643379 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:31.643726 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.643709 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:28:31.673465 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.673422 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" podStartSLOduration=9.04458462 podStartE2EDuration="27.673407028s" totalImagesPullingTime="18.628822408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:31.671537559 +0000 UTC m=+27.670323379" watchObservedRunningTime="2026-07-27 17:28:31.673407028 +0000 UTC m=+27.672192849" Jul 27 17:28:31.690307 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:31.690263 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zqfc" podStartSLOduration=3.956787355 podStartE2EDuration="27.690250047s" totalImagesPullingTime="23.733462692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:31.689790528 +0000 UTC m=+27.688576359" watchObservedRunningTime="2026-07-27 17:28:31.690250047 +0000 UTC m=+27.689035867" Jul 27 17:28:32.501016 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:32.500989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:32.501426 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:32.501085 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:33.285220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:33.285187 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2cq5f"] Jul 27 17:28:33.285454 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:33.285313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:33.285454 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:33.285419 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:33.288433 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:33.288398 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6gcqw"] Jul 27 17:28:33.288576 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:33.288511 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:33.288725 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:33.288678 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:33.289212 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:33.289191 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9rd7"] Jul 27 17:28:33.289290 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:33.289278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:33.289378 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:33.289362 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:34.443675 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:34.443588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:34.508454 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:34.508423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:34.639808 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:34.639777 2564 generic.go:356] "Generic (PLEG): container finished" podID="74acacfd-72eb-4498-9437-385f6d5d5ec2" containerID="2697a7bc32855a713b696d45c621690d641eec84f61ff6de6d6440f75bbbfc66" exitCode=0 Jul 27 17:28:34.639934 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:34.639826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerDied","Data":"2697a7bc32855a713b696d45c621690d641eec84f61ff6de6d6440f75bbbfc66"} Jul 27 17:28:35.499127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:35.498946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:35.499536 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:35.498949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:35.499536 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:35.498973 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:35.499536 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:35.499229 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:35.499536 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:35.499327 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:35.499536 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:35.499393 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:35.644421 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:35.644388 2564 generic.go:356] "Generic (PLEG): container finished" podID="74acacfd-72eb-4498-9437-385f6d5d5ec2" containerID="b12a2f83d04f9da8eee8923f00d5b952dca8509001529e10d8a16e78f4d41e45" exitCode=0 Jul 27 17:28:35.644579 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:35.644459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerDied","Data":"b12a2f83d04f9da8eee8923f00d5b952dca8509001529e10d8a16e78f4d41e45"} Jul 27 17:28:36.152683 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:36.152643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:36.208966 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:36.208928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:36.361554 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:36.361522 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:36.361732 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:36.361666 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 17:28:36.362234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:36.362209 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4zv6d" Jul 27 17:28:36.428053 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:36.427975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:37.498157 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:37.498105 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:37.498727 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:37.498262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:37.498727 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:37.498265 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gcqw" podUID="861d178f-ae27-4311-83a0-e0e3a9046297" Jul 27 17:28:37.498727 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:37.498301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:37.498727 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:37.498409 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cq5f" podUID="b53269e6-9bf1-4b56-8c80-73a3e3f630dd" Jul 27 17:28:37.498727 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:37.498491 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9rd7" podUID="736e470f-079a-4aea-b75c-c4ac3cf684bf" Jul 27 17:28:38.102585 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.102552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:38.102748 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.102725 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:38.102812 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.102800 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs podName:861d178f-ae27-4311-83a0-e0e3a9046297 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:10.102783841 +0000 UTC m=+66.101569640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs") pod "network-metrics-daemon-6gcqw" (UID: "861d178f-ae27-4311-83a0-e0e3a9046297") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:38.203441 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.203358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:38.203584 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.203478 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:38.203584 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.203532 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret podName:b53269e6-9bf1-4b56-8c80-73a3e3f630dd nodeName:}" failed. No retries permitted until 2026-07-27 17:29:10.203518136 +0000 UTC m=+66.202303936 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret") pod "global-pull-secret-syncer-2cq5f" (UID: "b53269e6-9bf1-4b56-8c80-73a3e3f630dd") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:38.303910 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.303871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:38.304075 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.304036 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:38.304075 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.304060 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:38.304075 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.304073 2564 projected.go:196] Error preparing data for projected volume kube-api-access-c2wfv for pod openshift-network-diagnostics/network-check-target-w9rd7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:38.304257 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.304134 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv podName:736e470f-079a-4aea-b75c-c4ac3cf684bf nodeName:}" failed. No retries permitted until 2026-07-27 17:29:10.304116629 +0000 UTC m=+66.302902443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2wfv" (UniqueName: "kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv") pod "network-check-target-w9rd7" (UID: "736e470f-079a-4aea-b75c-c4ac3cf684bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:38.354532 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.354502 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeReady" Jul 27 17:28:38.354697 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.354657 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 17:28:38.396479 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.396441 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-674664878d-4pqch"] Jul 27 17:28:38.398994 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.398971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.402650 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.402625 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 17:28:38.402871 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.402828 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 17:28:38.402981 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.402942 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lcj8p\"" Jul 27 17:28:38.403422 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.403404 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 17:28:38.411197 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.411175 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 17:28:38.416528 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.416506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674664878d-4pqch"] Jul 27 17:28:38.417192 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.417171 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xmskc"] Jul 27 17:28:38.419159 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.419126 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w2hvc"] Jul 27 17:28:38.419264 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.419249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.420932 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.420912 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.422284 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.422258 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 17:28:38.422483 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.422462 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 17:28:38.422754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.422735 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 17:28:38.422840 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.422808 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c5l2l\"" Jul 27 17:28:38.423390 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.423307 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 17:28:38.423519 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.423504 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 17:28:38.423639 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.423626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x9hw5\"" Jul 27 17:28:38.431424 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.431405 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xmskc"] Jul 27 17:28:38.441054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.438864 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w2hvc"] Jul 27 17:28:38.505251 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-installation-pull-secrets\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-bound-sa-token\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-trusted-ca\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/408b8bb0-3549-4640-b502-6ab8c6c97fec-tmp-dir\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfj9s\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-kube-api-access-sfj9s\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4354227a-bd56-4d80-b591-fa7e964b1759-cert\") pod \"ingress-canary-xmskc\" (UID: \"4354227a-bd56-4d80-b591-fa7e964b1759\") " pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-ca-trust-extracted\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-image-registry-private-configuration\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gwv\" (UniqueName: \"kubernetes.io/projected/408b8bb0-3549-4640-b502-6ab8c6c97fec-kube-api-access-c8gwv\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-certificates\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzb95\" (UniqueName: \"kubernetes.io/projected/4354227a-bd56-4d80-b591-fa7e964b1759-kube-api-access-lzb95\") pod \"ingress-canary-xmskc\" (UID: \"4354227a-bd56-4d80-b591-fa7e964b1759\") " pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.505662 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.505631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/408b8bb0-3549-4640-b502-6ab8c6c97fec-config-volume\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.606565 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-image-registry-private-configuration\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.606743 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gwv\" (UniqueName: \"kubernetes.io/projected/408b8bb0-3549-4640-b502-6ab8c6c97fec-kube-api-access-c8gwv\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.606743 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-certificates\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.606743 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzb95\" (UniqueName: \"kubernetes.io/projected/4354227a-bd56-4d80-b591-fa7e964b1759-kube-api-access-lzb95\") pod \"ingress-canary-xmskc\" (UID: \"4354227a-bd56-4d80-b591-fa7e964b1759\") " pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.606743 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/408b8bb0-3549-4640-b502-6ab8c6c97fec-config-volume\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.606965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-installation-pull-secrets\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.606965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-bound-sa-token\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.606965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-trusted-ca\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.606965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/408b8bb0-3549-4640-b502-6ab8c6c97fec-tmp-dir\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.606965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.606965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfj9s\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-kube-api-access-sfj9s\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.606965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4354227a-bd56-4d80-b591-fa7e964b1759-cert\") pod \"ingress-canary-xmskc\" (UID: \"4354227a-bd56-4d80-b591-fa7e964b1759\") " pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.607321 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.606976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.607321 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.607003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-ca-trust-extracted\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.607321 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.607287 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 17:28:38.607321 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.607305 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674664878d-4pqch: secret "image-registry-tls" not found Jul 27 17:28:38.607518 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.607363 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-ca-trust-extracted\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.607518 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.607388 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls podName:772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:39.107355359 +0000 UTC m=+35.106141160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls") pod "image-registry-674664878d-4pqch" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8") : secret "image-registry-tls" not found Jul 27 17:28:38.607518 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.607381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/408b8bb0-3549-4640-b502-6ab8c6c97fec-config-volume\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.607518 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.607484 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-certificates\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.607725 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.607566 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:38.607725 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:38.607633 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls podName:408b8bb0-3549-4640-b502-6ab8c6c97fec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:39.107613968 +0000 UTC m=+35.106399768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls") pod "dns-default-w2hvc" (UID: "408b8bb0-3549-4640-b502-6ab8c6c97fec") : secret "dns-default-metrics-tls" not found Jul 27 17:28:38.607837 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.607785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/408b8bb0-3549-4640-b502-6ab8c6c97fec-tmp-dir\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.608177 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.608068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-trusted-ca\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.611503 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.611477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-installation-pull-secrets\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.611608 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.611477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-image-registry-private-configuration\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.611719 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.611702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4354227a-bd56-4d80-b591-fa7e964b1759-cert\") pod \"ingress-canary-xmskc\" (UID: \"4354227a-bd56-4d80-b591-fa7e964b1759\") " pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.617263 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.616963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfj9s\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-kube-api-access-sfj9s\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.618378 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.618351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-bound-sa-token\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:38.618725 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.618682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzb95\" (UniqueName: \"kubernetes.io/projected/4354227a-bd56-4d80-b591-fa7e964b1759-kube-api-access-lzb95\") pod \"ingress-canary-xmskc\" (UID: \"4354227a-bd56-4d80-b591-fa7e964b1759\") " pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.619183 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.619162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gwv\" (UniqueName: \"kubernetes.io/projected/408b8bb0-3549-4640-b502-6ab8c6c97fec-kube-api-access-c8gwv\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:38.634935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.634902 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:38.730363 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.730320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xmskc" Jul 27 17:28:38.735453 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:38.735425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:39.111276 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.111243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:39.111459 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.111287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:39.111459 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.111422 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 17:28:39.111459 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.111447 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674664878d-4pqch: secret "image-registry-tls" not found Jul 27 17:28:39.111459 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.111455 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:39.111626 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.111505 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls podName:772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:40.111486555 +0000 UTC m=+36.110272354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls") pod "image-registry-674664878d-4pqch" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8") : secret "image-registry-tls" not found Jul 27 17:28:39.111626 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.111524 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls podName:408b8bb0-3549-4640-b502-6ab8c6c97fec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:40.111515328 +0000 UTC m=+36.110301127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls") pod "dns-default-w2hvc" (UID: "408b8bb0-3549-4640-b502-6ab8c6c97fec") : secret "dns-default-metrics-tls" not found Jul 27 17:28:39.222862 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.222814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:39.298025 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.297990 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk"] Jul 27 17:28:39.301951 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.301927 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" Jul 27 17:28:39.306788 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.306748 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 27 17:28:39.306925 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.306748 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5kxg5\"" Jul 27 17:28:39.307919 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.307901 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 27 17:28:39.413664 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.413596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4lh\" (UniqueName: \"kubernetes.io/projected/da828fde-51c6-4ce5-b59a-a40dafb28655-kube-api-access-8r4lh\") pod \"migrator-6865c9967-7w6rk\" (UID: \"da828fde-51c6-4ce5-b59a-a40dafb28655\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" Jul 27 17:28:39.421503 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.421464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:39.426294 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.426268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk"] Jul 27 17:28:39.498706 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.498538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:28:39.498805 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.498576 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:28:39.498954 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.498598 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:28:39.505073 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.504912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 17:28:39.505073 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.504942 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 17:28:39.505073 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.504982 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 17:28:39.505073 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.505016 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sz2v\"" Jul 27 17:28:39.505819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.505251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 17:28:39.505819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.505400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcn4g\"" Jul 27 17:28:39.515420 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.514783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4lh\" (UniqueName: \"kubernetes.io/projected/da828fde-51c6-4ce5-b59a-a40dafb28655-kube-api-access-8r4lh\") pod \"migrator-6865c9967-7w6rk\" (UID: \"da828fde-51c6-4ce5-b59a-a40dafb28655\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" Jul 27 17:28:39.521004 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.520983 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xmskc"] Jul 27 17:28:39.524779 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:39.524753 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4354227a_bd56_4d80_b591_fa7e964b1759.slice/crio-ecd390a716003dce283b892944c3c33b2f5b91fff896175476af983aeda411e6 WatchSource:0}: Error finding container ecd390a716003dce283b892944c3c33b2f5b91fff896175476af983aeda411e6: Status 404 returned error can't find the container with id ecd390a716003dce283b892944c3c33b2f5b91fff896175476af983aeda411e6 Jul 27 17:28:39.535277 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.535257 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4lh\" (UniqueName: \"kubernetes.io/projected/da828fde-51c6-4ce5-b59a-a40dafb28655-kube-api-access-8r4lh\") pod \"migrator-6865c9967-7w6rk\" (UID: \"da828fde-51c6-4ce5-b59a-a40dafb28655\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" Jul 27 17:28:39.569986 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.569962 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:39.583585 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.583562 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-75nxl"] Jul 27 17:28:39.589649 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.589634 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.599947 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.599920 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 17:28:39.600362 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.600345 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 17:28:39.600503 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.600486 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vzxqf\"" Jul 27 17:28:39.600552 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.600494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 17:28:39.608902 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.608884 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 17:28:39.612771 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.612750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" Jul 27 17:28:39.617024 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.617001 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-75nxl"] Jul 27 17:28:39.657403 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.657365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerStarted","Data":"7973aab64edb925dd12ac7a31a49a736b5111bd9d6eca1a625948308e03b7923"} Jul 27 17:28:39.658760 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.658728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xmskc" event={"ID":"4354227a-bd56-4d80-b591-fa7e964b1759","Type":"ContainerStarted","Data":"ecd390a716003dce283b892944c3c33b2f5b91fff896175476af983aeda411e6"} Jul 27 17:28:39.717006 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.716977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6756dc88-971f-4e21-b2d9-aac9389f8b74-data-volume\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.717127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.717008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.717127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.717103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.717213 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.717169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6756dc88-971f-4e21-b2d9-aac9389f8b74-crio-socket\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.717260 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.717212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj565\" (UniqueName: \"kubernetes.io/projected/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-api-access-lj565\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.730798 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.730773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:39.755255 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.755229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk"] Jul 27 17:28:39.758846 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:39.758810 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda828fde_51c6_4ce5_b59a_a40dafb28655.slice/crio-04b9a60df2713320b1fac42586cb82ddae693df1ce6bcb7719d511b41022e342 WatchSource:0}: Error finding container 04b9a60df2713320b1fac42586cb82ddae693df1ce6bcb7719d511b41022e342: Status 404 returned error can't find the container with id 04b9a60df2713320b1fac42586cb82ddae693df1ce6bcb7719d511b41022e342 Jul 27 17:28:39.818538 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.818507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.818695 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.818548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6756dc88-971f-4e21-b2d9-aac9389f8b74-crio-socket\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.818695 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.818632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj565\" (UniqueName: \"kubernetes.io/projected/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-api-access-lj565\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.818695 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.818643 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:39.818824 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.818752 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:40.318730802 +0000 UTC m=+36.317516602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : configmap "kube-rbac-proxy" not found Jul 27 17:28:39.818824 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.818803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6756dc88-971f-4e21-b2d9-aac9389f8b74-data-volume\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.818922 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.818833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.818965 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.818924 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6756dc88-971f-4e21-b2d9-aac9389f8b74-crio-socket\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.819081 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.819065 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:39.819129 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.819110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6756dc88-971f-4e21-b2d9-aac9389f8b74-data-volume\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.819232 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:39.819162 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:40.319124281 +0000 UTC m=+36.317910089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:39.832263 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.832246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj565\" (UniqueName: \"kubernetes.io/projected/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-api-access-lj565\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:39.985900 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:39.985867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:40.093935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.093882 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:40.121594 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.121553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:40.121754 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.121675 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:40.121754 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.121740 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls podName:408b8bb0-3549-4640-b502-6ab8c6c97fec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.121722829 +0000 UTC m=+38.120508628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls") pod "dns-default-w2hvc" (UID: "408b8bb0-3549-4640-b502-6ab8c6c97fec") : secret "dns-default-metrics-tls" not found Jul 27 17:28:40.121874 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.121853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:40.122021 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.121943 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 17:28:40.122021 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.121960 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674664878d-4pqch: secret "image-registry-tls" not found Jul 27 17:28:40.122021 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.122002 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls podName:772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.12199141 +0000 UTC m=+38.120777209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls") pod "image-registry-674664878d-4pqch" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8") : secret "image-registry-tls" not found Jul 27 17:28:40.323701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.323609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:40.323877 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.323750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:40.323877 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.323750 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:40.323877 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.323828 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:40.323877 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.323849 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:41.323828328 +0000 UTC m=+37.322614147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : configmap "kube-rbac-proxy" not found Jul 27 17:28:40.324029 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:40.323885 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:41.32386846 +0000 UTC m=+37.322654271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:40.662511 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.662431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" event={"ID":"da828fde-51c6-4ce5-b59a-a40dafb28655","Type":"ContainerStarted","Data":"04b9a60df2713320b1fac42586cb82ddae693df1ce6bcb7719d511b41022e342"} Jul 27 17:28:40.665564 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.665536 2564 generic.go:356] "Generic (PLEG): container finished" podID="74acacfd-72eb-4498-9437-385f6d5d5ec2" containerID="7973aab64edb925dd12ac7a31a49a736b5111bd9d6eca1a625948308e03b7923" exitCode=0 Jul 27 17:28:40.665721 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:40.665585 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerDied","Data":"7973aab64edb925dd12ac7a31a49a736b5111bd9d6eca1a625948308e03b7923"} Jul 27 17:28:41.333293 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.332964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:41.333468 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.333357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:41.333468 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:41.333112 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:41.333468 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:41.333450 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:41.333468 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:41.333457 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.333431702 +0000 UTC m=+39.332217512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:41.333697 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:41.333512 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.333483111 +0000 UTC m=+39.332268910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : configmap "kube-rbac-proxy" not found Jul 27 17:28:41.600760 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.600674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-v66bg"] Jul 27 17:28:41.607133 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.607105 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.613316 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.613280 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 17:28:41.613476 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.613357 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 17:28:41.614194 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.613996 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 17:28:41.614902 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.614879 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 17:28:41.615036 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.615014 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7ztf2\"" Jul 27 17:28:41.631037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.631010 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-v66bg"] Jul 27 17:28:41.672219 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.672121 2564 generic.go:356] "Generic (PLEG): container finished" podID="74acacfd-72eb-4498-9437-385f6d5d5ec2" containerID="90ab0977116658c5f0d8aaade30316d0b13a51298bbff21c43b62ff4ab729b12" exitCode=0 Jul 27 17:28:41.672219 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.672183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerDied","Data":"90ab0977116658c5f0d8aaade30316d0b13a51298bbff21c43b62ff4ab729b12"} Jul 27 17:28:41.737006 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.736973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-signing-key\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.737180 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.737036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zljf\" (UniqueName: \"kubernetes.io/projected/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-kube-api-access-4zljf\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.737180 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.737091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-signing-cabundle\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.758040 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.758017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dsrdc_6d3767a9-6ba7-481d-b6bf-b2f76eb5a679/dns-node-resolver/0.log" Jul 27 17:28:41.838114 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.838076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-signing-key\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.838291 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.838165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zljf\" (UniqueName: \"kubernetes.io/projected/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-kube-api-access-4zljf\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.838291 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.838215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-signing-cabundle\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.839061 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.839036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-signing-cabundle\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.841826 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.841801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-signing-key\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.851623 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.851562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zljf\" (UniqueName: \"kubernetes.io/projected/d31135b5-4d36-4fa6-a597-e5f72ec0f42a-kube-api-access-4zljf\") pod \"service-ca-658d7b7c78-v66bg\" (UID: \"d31135b5-4d36-4fa6-a597-e5f72ec0f42a\") " pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.917660 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.917616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" Jul 27 17:28:41.931024 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:41.930846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:42.106425 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.106361 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:42.141510 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.141479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:42.141664 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.141519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:42.141664 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:42.141637 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 17:28:42.141664 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:42.141659 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674664878d-4pqch: secret "image-registry-tls" not found Jul 27 17:28:42.141822 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:42.141715 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls podName:772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:46.141698336 +0000 UTC m=+42.140484155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls") pod "image-registry-674664878d-4pqch" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8") : secret "image-registry-tls" not found Jul 27 17:28:42.141822 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:42.141638 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:42.141822 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:42.141797 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls podName:408b8bb0-3549-4640-b502-6ab8c6c97fec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:46.1417791 +0000 UTC m=+42.140564919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls") pod "dns-default-w2hvc" (UID: "408b8bb0-3549-4640-b502-6ab8c6c97fec") : secret "dns-default-metrics-tls" not found Jul 27 17:28:42.277366 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.277273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:42.373860 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.373618 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-v66bg"] Jul 27 17:28:42.377635 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:42.377607 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31135b5_4d36_4fa6_a597_e5f72ec0f42a.slice/crio-b54a0ad4c0fd621fadaddbdc95cda9aa19380fdd1aff3361dcf58ed88ac64e73 WatchSource:0}: Error finding container b54a0ad4c0fd621fadaddbdc95cda9aa19380fdd1aff3361dcf58ed88ac64e73: Status 404 returned error can't find the container with id b54a0ad4c0fd621fadaddbdc95cda9aa19380fdd1aff3361dcf58ed88ac64e73 Jul 27 17:28:42.546719 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.546652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:42.676624 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.676535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" event={"ID":"da828fde-51c6-4ce5-b59a-a40dafb28655","Type":"ContainerStarted","Data":"703ba1322bb077c407d023beb789e0260b975015bcbfb7b259a9a6867f921e68"} Jul 27 17:28:42.676624 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.676597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" event={"ID":"da828fde-51c6-4ce5-b59a-a40dafb28655","Type":"ContainerStarted","Data":"3c49d3daea20f89ad6a8bd47dfa7617314de78d0ac3932ea6cb82f2d51b18e1b"} Jul 27 17:28:42.679943 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.679912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" event={"ID":"74acacfd-72eb-4498-9437-385f6d5d5ec2","Type":"ContainerStarted","Data":"9d8eac4950d84a2f7da72780a269cfced50e30832e59f10d2a196ec48cc99da1"} Jul 27 17:28:42.681479 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.681458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xmskc" event={"ID":"4354227a-bd56-4d80-b591-fa7e964b1759","Type":"ContainerStarted","Data":"e183c46d057ce0c42f9e196966fa7350d2b7d237f82059992330132f4c81236c"} Jul 27 17:28:42.682551 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.682527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" event={"ID":"d31135b5-4d36-4fa6-a597-e5f72ec0f42a","Type":"ContainerStarted","Data":"b54a0ad4c0fd621fadaddbdc95cda9aa19380fdd1aff3361dcf58ed88ac64e73"} Jul 27 17:28:42.699790 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.699745 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-7w6rk" podStartSLOduration=1.2471228540000001 podStartE2EDuration="3.699732796s" totalImagesPullingTime="2.452609942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:42.698970245 +0000 UTC m=+38.697756086" watchObservedRunningTime="2026-07-27 17:28:42.699732796 +0000 UTC m=+38.698518655" Jul 27 17:28:42.738448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.738409 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xkvfz" podStartSLOduration=8.641040308000001 podStartE2EDuration="38.738394328s" totalImagesPullingTime="30.09735402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:42.73593674 +0000 UTC m=+38.734722583" watchObservedRunningTime="2026-07-27 17:28:42.738394328 +0000 UTC m=+38.737180149" Jul 27 17:28:42.761331 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:42.761291 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xmskc" podStartSLOduration=2.075232096 podStartE2EDuration="4.761277586s" totalImagesPullingTime="2.68604549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:42.760651776 +0000 UTC m=+38.759437597" watchObservedRunningTime="2026-07-27 17:28:42.761277586 +0000 UTC m=+38.760063406" Jul 27 17:28:43.160819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:43.160787 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvbv6_4be3196f-ed07-462b-a955-e78df488a631/node-ca/0.log" Jul 27 17:28:43.350580 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:43.350545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:43.350775 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:43.350647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:43.350775 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:43.350697 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:43.350775 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:43.350766 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:43.350894 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:43.350785 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:47.350763448 +0000 UTC m=+43.349549249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : configmap "kube-rbac-proxy" not found Jul 27 17:28:43.350894 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:43.350814 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:47.350798408 +0000 UTC m=+43.349584265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:43.701129 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:43.701036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:43.758719 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:43.758639 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xmskc_4354227a-bd56-4d80-b591-fa7e964b1759/serve-healthcheck-canary/0.log" Jul 27 17:28:43.809428 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:43.809392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:43.957343 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:43.957264 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-7w6rk_da828fde-51c6-4ce5-b59a-a40dafb28655/migrator/0.log" Jul 27 17:28:44.156761 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:44.156733 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-7w6rk_da828fde-51c6-4ce5-b59a-a40dafb28655/graceful-termination/0.log" Jul 27 17:28:44.689424 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:44.689384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" event={"ID":"d31135b5-4d36-4fa6-a597-e5f72ec0f42a","Type":"ContainerStarted","Data":"71d48cbc8525aa1b0616d2e3f4678eb9a9020b833774761a8d11d7ef8ea5a797"} Jul 27 17:28:44.708542 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:44.708500 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-v66bg" podStartSLOduration=1.627103739 podStartE2EDuration="3.708486181s" totalImagesPullingTime="2.081382442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:44.707728821 +0000 UTC m=+40.706514633" watchObservedRunningTime="2026-07-27 17:28:44.708486181 +0000 UTC m=+40.707271981" Jul 27 17:28:45.553028 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:45.552993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:45.721706 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:45.721670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:45.822958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:45.822884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:45.925090 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:45.925038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:46.048848 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:46.048810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:46.177254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:46.177154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:46.177254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:46.177220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:46.177412 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:46.177276 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 17:28:46.177412 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:46.177301 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674664878d-4pqch: secret "image-registry-tls" not found Jul 27 17:28:46.177412 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:46.177362 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:46.177412 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:46.177374 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls podName:772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:54.177351289 +0000 UTC m=+50.176137101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls") pod "image-registry-674664878d-4pqch" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8") : secret "image-registry-tls" not found Jul 27 17:28:46.177412 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:46.177409 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls podName:408b8bb0-3549-4640-b502-6ab8c6c97fec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:54.177395162 +0000 UTC m=+50.176180972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls") pod "dns-default-w2hvc" (UID: "408b8bb0-3549-4640-b502-6ab8c6c97fec") : secret "dns-default-metrics-tls" not found Jul 27 17:28:47.387941 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:47.387901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:47.388376 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:47.388020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:47.388376 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:47.388024 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:47.388376 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:47.388097 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:47.388376 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:47.388114 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:55.388095221 +0000 UTC m=+51.386881035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : configmap "kube-rbac-proxy" not found Jul 27 17:28:47.388376 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:28:47.388136 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls podName:6756dc88-971f-4e21-b2d9-aac9389f8b74 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:55.388125513 +0000 UTC m=+51.386911327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-75nxl" (UID: "6756dc88-971f-4e21-b2d9-aac9389f8b74") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:47.443314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:47.443273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:47.520900 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:47.520865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:47.856672 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:47.856641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:47.976070 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:47.976036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:48.108470 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:48.108391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:48.643293 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:48.643261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:48.874279 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:48.874248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:49.055290 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:49.055253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:49.361385 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:49.361311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:49.463382 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:49.463353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:49.662512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:49.662427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:50.034344 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:50.034311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:50.140997 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:50.140962 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:50.443617 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:50.443536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:50.836336 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:50.836307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:51.331878 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:51.331842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:51.429928 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:51.429898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:51.545659 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:51.545629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:51.649189 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:51.649164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:52.545359 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:52.545327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:52.649691 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:52.649659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:53.936161 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:53.936114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:53.993771 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:53.993740 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:54.050486 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.050459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:54.242921 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.242886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:54.244942 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.244923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:54.245008 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.244955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:54.248764 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.248737 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"image-registry-674664878d-4pqch\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:54.249088 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.249066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/408b8bb0-3549-4640-b502-6ab8c6c97fec-metrics-tls\") pod \"dns-default-w2hvc\" (UID: \"408b8bb0-3549-4640-b502-6ab8c6c97fec\") " pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:54.311549 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.311515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:54.322985 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.322959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:54.338657 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.338631 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:54.472208 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.472177 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674664878d-4pqch"] Jul 27 17:28:54.476686 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:54.476648 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772f3d3c_60c2_4e11_8a26_ee8ae03ec5a8.slice/crio-a22799895608f683317d15e07db3b7e6e90819495434d72befcf21dc0e70596a WatchSource:0}: Error finding container a22799895608f683317d15e07db3b7e6e90819495434d72befcf21dc0e70596a: Status 404 returned error can't find the container with id a22799895608f683317d15e07db3b7e6e90819495434d72befcf21dc0e70596a Jul 27 17:28:54.505436 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.505408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w2hvc"] Jul 27 17:28:54.508434 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:54.508407 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408b8bb0_3549_4640_b502_6ab8c6c97fec.slice/crio-bfbe44b1e89128237c79b2652c2f94022310ce366dc57aa81fdc118043610009 WatchSource:0}: Error finding container bfbe44b1e89128237c79b2652c2f94022310ce366dc57aa81fdc118043610009: Status 404 returned error can't find the container with id bfbe44b1e89128237c79b2652c2f94022310ce366dc57aa81fdc118043610009 Jul 27 17:28:54.716106 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.716069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w2hvc" event={"ID":"408b8bb0-3549-4640-b502-6ab8c6c97fec","Type":"ContainerStarted","Data":"bfbe44b1e89128237c79b2652c2f94022310ce366dc57aa81fdc118043610009"} Jul 27 17:28:54.717585 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.717552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674664878d-4pqch" event={"ID":"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8","Type":"ContainerStarted","Data":"5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7"} Jul 27 17:28:54.717733 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.717600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674664878d-4pqch" event={"ID":"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8","Type":"ContainerStarted","Data":"a22799895608f683317d15e07db3b7e6e90819495434d72befcf21dc0e70596a"} Jul 27 17:28:54.717733 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.717698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:28:54.740271 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:54.737124 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-674664878d-4pqch" podStartSLOduration=20.737102597 podStartE2EDuration="20.737102597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:54.736013224 +0000 UTC m=+50.734799044" watchObservedRunningTime="2026-07-27 17:28:54.737102597 +0000 UTC m=+50.735888419" Jul 27 17:28:55.247574 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:55.247535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:55.454850 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:55.454815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:55.455025 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:55.454960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:55.455473 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:55.455448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6756dc88-971f-4e21-b2d9-aac9389f8b74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:55.457896 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:55.457873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6756dc88-971f-4e21-b2d9-aac9389f8b74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-75nxl\" (UID: \"6756dc88-971f-4e21-b2d9-aac9389f8b74\") " pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:55.498676 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:55.498607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-75nxl" Jul 27 17:28:55.645037 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:55.645006 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-75nxl"] Jul 27 17:28:56.157724 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:28:56.157672 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6756dc88_971f_4e21_b2d9_aac9389f8b74.slice/crio-f867f2af6f7b8097e0e1b87708d8cc6d3db20e70d102186c899cd394ca620810 WatchSource:0}: Error finding container f867f2af6f7b8097e0e1b87708d8cc6d3db20e70d102186c899cd394ca620810: Status 404 returned error can't find the container with id f867f2af6f7b8097e0e1b87708d8cc6d3db20e70d102186c899cd394ca620810 Jul 27 17:28:56.367113 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:56.367071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:56.467279 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:56.467253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:28:56.726750 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:56.726711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w2hvc" event={"ID":"408b8bb0-3549-4640-b502-6ab8c6c97fec","Type":"ContainerStarted","Data":"8e867a08cf9934ffa9f2f31d0fde5a68762682519f237ee5026248ebffb7df4f"} Jul 27 17:28:56.726924 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:56.726782 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w2hvc" event={"ID":"408b8bb0-3549-4640-b502-6ab8c6c97fec","Type":"ContainerStarted","Data":"187f6c0bff5bd7230b80a7cfd6c4f0a0ab069611d96053df99b45f2f8154581b"} Jul 27 17:28:56.726924 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:56.726903 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w2hvc" Jul 27 17:28:56.728572 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:56.728505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-75nxl" event={"ID":"6756dc88-971f-4e21-b2d9-aac9389f8b74","Type":"ContainerStarted","Data":"f867f2af6f7b8097e0e1b87708d8cc6d3db20e70d102186c899cd394ca620810"} Jul 27 17:28:56.746742 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:28:56.746694 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w2hvc" podStartSLOduration=16.798847513 podStartE2EDuration="18.746679383s" totalImagesPullingTime="1.94783187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:56.745133 +0000 UTC m=+52.743918822" watchObservedRunningTime="2026-07-27 17:28:56.746679383 +0000 UTC m=+52.745465181" Jul 27 17:29:01.276222 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:01.276134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:01.388484 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:01.388447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:01.466219 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:01.466190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:01.579121 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:01.579041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:01.749637 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:01.749597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-75nxl" event={"ID":"6756dc88-971f-4e21-b2d9-aac9389f8b74","Type":"ContainerStarted","Data":"058bdcd7dbe132cca3db6f29504941903e2805f1a1c6e9ef407c4974e7da9402"} Jul 27 17:29:01.941940 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:01.941839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:03.474220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:03.474183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:03.537418 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:03.537367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:03.657343 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:03.657317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r948w" Jul 27 17:29:04.404717 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:04.404680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:04.527967 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:04.527929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:04.665632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:04.665574 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674664878d-4pqch"] Jul 27 17:29:04.761843 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:04.761810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-75nxl" event={"ID":"6756dc88-971f-4e21-b2d9-aac9389f8b74","Type":"ContainerStarted","Data":"5f17cda29f0a54c5a9579d6f0460d0df4679773b820f5956f381cbbb9d3ed489"} Jul 27 17:29:04.765830 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:04.765723 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:04.958042 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:04.957960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:05.060182 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:05.060131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:05.154295 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:05.154258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:06.555594 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:06.555556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:06.618166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:06.618126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:06.734783 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:06.734755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w2hvc" Jul 27 17:29:07.438100 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:07.438065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:07.496193 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:07.496162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:10.180337 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.180313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:29:10.183557 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.183537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 17:29:10.192886 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.192866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/861d178f-ae27-4311-83a0-e0e3a9046297-metrics-certs\") pod \"network-metrics-daemon-6gcqw\" (UID: \"861d178f-ae27-4311-83a0-e0e3a9046297\") " pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:29:10.281688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.281658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:29:10.284415 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.284395 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 17:29:10.294766 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.294743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b53269e6-9bf1-4b56-8c80-73a3e3f630dd-original-pull-secret\") pod \"global-pull-secret-syncer-2cq5f\" (UID: \"b53269e6-9bf1-4b56-8c80-73a3e3f630dd\") " pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:29:10.382469 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.382429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:29:10.385255 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.385233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 17:29:10.395476 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.395450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 17:29:10.406032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.406004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wfv\" (UniqueName: \"kubernetes.io/projected/736e470f-079a-4aea-b75c-c4ac3cf684bf-kube-api-access-c2wfv\") pod \"network-check-target-w9rd7\" (UID: \"736e470f-079a-4aea-b75c-c4ac3cf684bf\") " pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:29:10.411810 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.411793 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cq5f" Jul 27 17:29:10.422687 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.422639 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcn4g\"" Jul 27 17:29:10.429458 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.429441 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sz2v\"" Jul 27 17:29:10.430463 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.430449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:29:10.437646 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.437630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gcqw" Jul 27 17:29:10.575040 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.574932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:10.584359 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.584051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2cq5f"] Jul 27 17:29:10.586740 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:29:10.586689 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53269e6_9bf1_4b56_8c80_73a3e3f630dd.slice/crio-f02a15a8c28c440936417b617e06d9b80ee7a1819fa503a8563887dce47121a0 WatchSource:0}: Error finding container f02a15a8c28c440936417b617e06d9b80ee7a1819fa503a8563887dce47121a0: Status 404 returned error can't find the container with id f02a15a8c28c440936417b617e06d9b80ee7a1819fa503a8563887dce47121a0 Jul 27 17:29:10.613215 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.613192 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9rd7"] Jul 27 17:29:10.616510 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:29:10.616486 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736e470f_079a_4aea_b75c_c4ac3cf684bf.slice/crio-71cb671cc1c5ba5bb7ab7d4dbaa4c2fcd566ce51abfca3c783064b27eb8cf3eb WatchSource:0}: Error finding container 71cb671cc1c5ba5bb7ab7d4dbaa4c2fcd566ce51abfca3c783064b27eb8cf3eb: Status 404 returned error can't find the container with id 71cb671cc1c5ba5bb7ab7d4dbaa4c2fcd566ce51abfca3c783064b27eb8cf3eb Jul 27 17:29:10.639392 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.639356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6gcqw"] Jul 27 17:29:10.641758 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:29:10.641733 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861d178f_ae27_4311_83a0_e0e3a9046297.slice/crio-318ef22b7e36964df6cf6c299bc4c907e633b51f3b75f4a04366b05eb0fcffbe WatchSource:0}: Error finding container 318ef22b7e36964df6cf6c299bc4c907e633b51f3b75f4a04366b05eb0fcffbe: Status 404 returned error can't find the container with id 318ef22b7e36964df6cf6c299bc4c907e633b51f3b75f4a04366b05eb0fcffbe Jul 27 17:29:10.644812 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.644791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:10.721401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.721324 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:10.783451 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.783412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-75nxl" event={"ID":"6756dc88-971f-4e21-b2d9-aac9389f8b74","Type":"ContainerStarted","Data":"386680e7dbd814e3af166bfa235fea8c8493e802ff5ef3b5f49a0ba758d196a4"} Jul 27 17:29:10.784583 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.784558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gcqw" event={"ID":"861d178f-ae27-4311-83a0-e0e3a9046297","Type":"ContainerStarted","Data":"318ef22b7e36964df6cf6c299bc4c907e633b51f3b75f4a04366b05eb0fcffbe"} Jul 27 17:29:10.785730 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.785705 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9rd7" event={"ID":"736e470f-079a-4aea-b75c-c4ac3cf684bf","Type":"ContainerStarted","Data":"71cb671cc1c5ba5bb7ab7d4dbaa4c2fcd566ce51abfca3c783064b27eb8cf3eb"} Jul 27 17:29:10.786831 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.786801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2cq5f" event={"ID":"b53269e6-9bf1-4b56-8c80-73a3e3f630dd","Type":"ContainerStarted","Data":"f02a15a8c28c440936417b617e06d9b80ee7a1819fa503a8563887dce47121a0"} Jul 27 17:29:10.804999 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.804954 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-75nxl" podStartSLOduration=17.891207912 podStartE2EDuration="31.804940976s" totalImagesPullingTime="13.913733064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:10.802730701 +0000 UTC m=+66.801516521" watchObservedRunningTime="2026-07-27 17:29:10.804940976 +0000 UTC m=+66.803726798" Jul 27 17:29:10.973114 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:10.973026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:11.576657 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:11.576598 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:11.643342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:11.643262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:13.690542 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:13.690501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:13.795816 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:13.795776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:14.670987 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:14.670957 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:29:15.131838 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.131802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:15.232541 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.232493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:15.376436 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.376360 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6458fc48fd-wzs5z"] Jul 27 17:29:15.378191 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.378174 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.381404 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.381380 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 17:29:15.381665 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.381644 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 17:29:15.382645 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.382628 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 17:29:15.382754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.382652 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c76kr\"" Jul 27 17:29:15.382754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.382737 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 17:29:15.383216 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.383199 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 17:29:15.383337 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.383319 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 17:29:15.383609 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.383592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 17:29:15.389367 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.389349 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 17:29:15.390708 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.390688 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6458fc48fd-wzs5z"] Jul 27 17:29:15.521818 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.521780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-oauth-serving-cert\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.522009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.521844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-config\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.522009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.521946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzds\" (UniqueName: \"kubernetes.io/projected/e2d65e1e-eab2-4393-aea6-497182ca8d4f-kube-api-access-9bzds\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.522127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.522009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-trusted-ca-bundle\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.522127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.522063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-service-ca\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.522127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.522099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-serving-cert\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.522311 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.522122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-oauth-config\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.622842 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.622807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-trusted-ca-bundle\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.623032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.622860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-service-ca\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.623032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.622889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-serving-cert\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.623032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.622905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-oauth-config\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.623032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.622933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-oauth-serving-cert\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.623032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.622973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-config\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.623032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.623011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzds\" (UniqueName: \"kubernetes.io/projected/e2d65e1e-eab2-4393-aea6-497182ca8d4f-kube-api-access-9bzds\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.624009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.623740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-service-ca\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.624009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.623950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-trusted-ca-bundle\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.624197 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.624074 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-oauth-serving-cert\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.624197 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.624109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-config\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.625888 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.625865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-serving-cert\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.625992 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.625872 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-oauth-config\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.632893 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.632842 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzds\" (UniqueName: \"kubernetes.io/projected/e2d65e1e-eab2-4393-aea6-497182ca8d4f-kube-api-access-9bzds\") pod \"console-6458fc48fd-wzs5z\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.686787 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.686760 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:15.807585 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.807553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9rd7" event={"ID":"736e470f-079a-4aea-b75c-c4ac3cf684bf","Type":"ContainerStarted","Data":"18d6507f5ddfd09d8909b7018d3924ffc2e9e377bccc4d1d012c9ec949b3ad06"} Jul 27 17:29:15.807783 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.807700 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:29:15.809269 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.809241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2cq5f" event={"ID":"b53269e6-9bf1-4b56-8c80-73a3e3f630dd","Type":"ContainerStarted","Data":"d0edd144a15efaf9d1ee9d524227ad7da83ebf2e82a6b23991219e4c9bb46131"} Jul 27 17:29:15.823824 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.823782 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w9rd7" podStartSLOduration=67.393438906 podStartE2EDuration="1m11.823769111s" totalImagesPullingTime="4.430330205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:15.823473326 +0000 UTC m=+71.822259171" watchObservedRunningTime="2026-07-27 17:29:15.823769111 +0000 UTC m=+71.822554909" Jul 27 17:29:15.838850 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.838805 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2cq5f" podStartSLOduration=67.375527513 podStartE2EDuration="1m11.838788773s" totalImagesPullingTime="4.46326126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:15.838132176 +0000 UTC m=+71.836918008" watchObservedRunningTime="2026-07-27 17:29:15.838788773 +0000 UTC m=+71.837574596" Jul 27 17:29:15.877781 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.877747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:15.978661 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:15.978628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:16.001221 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:16.001104 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6458fc48fd-wzs5z"] Jul 27 17:29:16.005130 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:29:16.005099 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d65e1e_eab2_4393_aea6_497182ca8d4f.slice/crio-f909a0502690c6824834e6123aa780c5d34fb3c070de2aef1ba2761153909251 WatchSource:0}: Error finding container f909a0502690c6824834e6123aa780c5d34fb3c070de2aef1ba2761153909251: Status 404 returned error can't find the container with id f909a0502690c6824834e6123aa780c5d34fb3c070de2aef1ba2761153909251 Jul 27 17:29:16.167002 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:16.166910 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:16.816188 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:16.816151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gcqw" event={"ID":"861d178f-ae27-4311-83a0-e0e3a9046297","Type":"ContainerStarted","Data":"d096e1e42dea05e567bdf7da4bae2fcc014ca09d1f954ef0810642f955e837d3"} Jul 27 17:29:16.816419 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:16.816219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gcqw" event={"ID":"861d178f-ae27-4311-83a0-e0e3a9046297","Type":"ContainerStarted","Data":"ff32b910798cf3e7a80037ffaace62f2b9ec7d73d3ca8c7aa03d872cb43285de"} Jul 27 17:29:16.818201 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:16.818167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6458fc48fd-wzs5z" event={"ID":"e2d65e1e-eab2-4393-aea6-497182ca8d4f","Type":"ContainerStarted","Data":"f909a0502690c6824834e6123aa780c5d34fb3c070de2aef1ba2761153909251"} Jul 27 17:29:16.833153 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:16.833092 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6gcqw" podStartSLOduration=67.57847307 podStartE2EDuration="1m12.833074502s" totalImagesPullingTime="5.254601432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:16.831500543 +0000 UTC m=+72.830286362" watchObservedRunningTime="2026-07-27 17:29:16.833074502 +0000 UTC m=+72.831860323" Jul 27 17:29:17.789959 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.789925 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-drtbp"] Jul 27 17:29:17.792096 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.792077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:17.794998 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.794972 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-49vjn\"" Jul 27 17:29:17.796335 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.796313 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 17:29:17.796444 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.796356 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 17:29:17.796820 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.796648 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 27 17:29:17.796820 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.796681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 17:29:17.796820 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.796708 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 27 17:29:17.803090 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.803023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-drtbp"] Jul 27 17:29:17.942503 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.942465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:17.942682 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.942549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176d2221-4acd-483e-9630-bebac5a413e8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:17.942682 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.942606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:17.942682 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:17.942666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6gs\" (UniqueName: \"kubernetes.io/projected/176d2221-4acd-483e-9630-bebac5a413e8-kube-api-access-bx6gs\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.043483 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.043392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.043483 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.043449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176d2221-4acd-483e-9630-bebac5a413e8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.043483 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.043484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.043752 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.043519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6gs\" (UniqueName: \"kubernetes.io/projected/176d2221-4acd-483e-9630-bebac5a413e8-kube-api-access-bx6gs\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.043752 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:29:18.043647 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 27 17:29:18.043752 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:29:18.043723 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-tls podName:176d2221-4acd-483e-9630-bebac5a413e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:18.543701991 +0000 UTC m=+74.542487803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-drtbp" (UID: "176d2221-4acd-483e-9630-bebac5a413e8") : secret "prometheus-operator-tls" not found Jul 27 17:29:18.044314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.044289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176d2221-4acd-483e-9630-bebac5a413e8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.046114 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.046085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.054727 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.054704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6gs\" (UniqueName: \"kubernetes.io/projected/176d2221-4acd-483e-9630-bebac5a413e8-kube-api-access-bx6gs\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.549829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.549783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.552281 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.552256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/176d2221-4acd-483e-9630-bebac5a413e8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-drtbp\" (UID: \"176d2221-4acd-483e-9630-bebac5a413e8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.633629 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.633596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:18.703859 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.703821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" Jul 27 17:29:18.731152 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.731097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:18.924547 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:18.924478 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-drtbp"] Jul 27 17:29:19.381497 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:19.381458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:19.479007 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:19.478970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:19.830709 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:19.830666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" event={"ID":"176d2221-4acd-483e-9630-bebac5a413e8","Type":"ContainerStarted","Data":"d545e3ba530279e9f29412b4caecaee3e372fc43a04188059902b591158f9054"} Jul 27 17:29:19.832440 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:19.832412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6458fc48fd-wzs5z" event={"ID":"e2d65e1e-eab2-4393-aea6-497182ca8d4f","Type":"ContainerStarted","Data":"50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06"} Jul 27 17:29:20.296357 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:20.296304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:20.837097 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:20.837062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" event={"ID":"176d2221-4acd-483e-9630-bebac5a413e8","Type":"ContainerStarted","Data":"e4efeee268735421202b611a58123d3d9db265d7294d5ab42633d43e337338b0"} Jul 27 17:29:20.837292 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:20.837121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" event={"ID":"176d2221-4acd-483e-9630-bebac5a413e8","Type":"ContainerStarted","Data":"306527368dd137891bdde1a0666ad413909f77d2992760f6462cf3c2273219c9"} Jul 27 17:29:20.857572 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:20.857526 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-drtbp" podStartSLOduration=2.773945088 podStartE2EDuration="3.85751283s" totalImagesPullingTime="1.083567742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:20.855981292 +0000 UTC m=+76.854767112" watchObservedRunningTime="2026-07-27 17:29:20.85751283 +0000 UTC m=+76.856298650" Jul 27 17:29:20.857855 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:20.857835 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6458fc48fd-wzs5z" podStartSLOduration=3.042796913 podStartE2EDuration="5.857829245s" totalImagesPullingTime="2.815032332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:19.852750703 +0000 UTC m=+75.851536545" watchObservedRunningTime="2026-07-27 17:29:20.857829245 +0000 UTC m=+76.856615066" Jul 27 17:29:20.978588 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:20.978560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:21.079697 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:21.079661 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:22.904351 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:22.904314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.134442 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.134407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.152538 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.152506 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mcrs9"] Jul 27 17:29:23.155162 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.155103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.157797 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.157774 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 17:29:23.157906 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.157773 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 17:29:23.158342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.158325 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 17:29:23.158399 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.158346 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fhlh7\"" Jul 27 17:29:23.232047 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.232022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.286866 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.286837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-wtmp\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287007 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.286875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287007 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.286926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-textfile\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287007 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.286955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-tls\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287007 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.286997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287228 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.287024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-root\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287228 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.287047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c29a54b-c791-496a-8947-6e68af57a0e8-metrics-client-ca\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287228 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.287071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-sys\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.287228 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.287104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzz4p\" (UniqueName: \"kubernetes.io/projected/5c29a54b-c791-496a-8947-6e68af57a0e8-kube-api-access-bzz4p\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.307458 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.307430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.387781 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-textfile\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.387881 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-tls\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.387881 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.387881 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-root\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388018 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c29a54b-c791-496a-8947-6e68af57a0e8-metrics-client-ca\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388018 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-sys\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388018 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:29:23.387930 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 27 17:29:23.388018 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzz4p\" (UniqueName: \"kubernetes.io/projected/5c29a54b-c791-496a-8947-6e68af57a0e8-kube-api-access-bzz4p\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388018 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-wtmp\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388018 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:29:23.387999 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-tls podName:5c29a54b-c791-496a-8947-6e68af57a0e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:23.887978012 +0000 UTC m=+79.886763812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-tls") pod "node-exporter-mcrs9" (UID: "5c29a54b-c791-496a-8947-6e68af57a0e8") : secret "node-exporter-tls" not found Jul 27 17:29:23.388018 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.387999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-root\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388373 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.388043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388373 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.388129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-wtmp\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388373 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.388200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c29a54b-c791-496a-8947-6e68af57a0e8-sys\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388373 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.388178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-textfile\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388567 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.388551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.388869 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.388848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c29a54b-c791-496a-8947-6e68af57a0e8-metrics-client-ca\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.390387 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.390368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.401667 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.401639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzz4p\" (UniqueName: \"kubernetes.io/projected/5c29a54b-c791-496a-8947-6e68af57a0e8-kube-api-access-bzz4p\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.429225 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.429164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.529560 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.529536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.630535 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.630503 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.746085 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.746057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:23.892481 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.892452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-tls\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:23.894742 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:23.894720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c29a54b-c791-496a-8947-6e68af57a0e8-node-exporter-tls\") pod \"node-exporter-mcrs9\" (UID: \"5c29a54b-c791-496a-8947-6e68af57a0e8\") " pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:24.065325 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:24.065244 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mcrs9" Jul 27 17:29:24.075191 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:29:24.075162 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c29a54b_c791_496a_8947_6e68af57a0e8.slice/crio-4ae58dfd9189220ed5453b2faadd09e4c92e2b069e152c2df6213f393a888074 WatchSource:0}: Error finding container 4ae58dfd9189220ed5453b2faadd09e4c92e2b069e152c2df6213f393a888074: Status 404 returned error can't find the container with id 4ae58dfd9189220ed5453b2faadd09e4c92e2b069e152c2df6213f393a888074 Jul 27 17:29:24.155014 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:24.154987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:24.238565 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:24.238532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:24.353917 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:24.353831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:24.798224 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:24.798193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:24.851944 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:24.851911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mcrs9" event={"ID":"5c29a54b-c791-496a-8947-6e68af57a0e8","Type":"ContainerStarted","Data":"4ae58dfd9189220ed5453b2faadd09e4c92e2b069e152c2df6213f393a888074"} Jul 27 17:29:25.687648 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:25.687619 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:25.688087 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:25.687921 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:25.692567 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:25.692542 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:25.856396 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:25.856371 2564 generic.go:356] "Generic (PLEG): container finished" podID="5c29a54b-c791-496a-8947-6e68af57a0e8" containerID="c7763f6d4799c690198a1551b8a5af7f4016434dc5aee7514e04fea5f1391183" exitCode=0 Jul 27 17:29:25.856502 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:25.856453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mcrs9" event={"ID":"5c29a54b-c791-496a-8947-6e68af57a0e8","Type":"ContainerDied","Data":"c7763f6d4799c690198a1551b8a5af7f4016434dc5aee7514e04fea5f1391183"} Jul 27 17:29:25.860597 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:25.860576 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:29:26.135516 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.135403 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b8b8448-gw5vz"] Jul 27 17:29:26.139249 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.139231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.141934 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.141909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 27 17:29:26.142041 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.141957 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 27 17:29:26.142041 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.141974 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 27 17:29:26.142041 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.141995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h55w6\"" Jul 27 17:29:26.142220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.142055 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-55vjto8q8rivp\"" Jul 27 17:29:26.142220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.142069 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 27 17:29:26.142404 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.142380 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 27 17:29:26.152570 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.152548 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b8b8448-gw5vz"] Jul 27 17:29:26.212185 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.212185 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.212408 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.212408 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-grpc-tls\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.212408 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d14aa51-1b66-4d61-9515-61edbf06d2cb-metrics-client-ca\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.212408 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.212408 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjdx\" (UniqueName: \"kubernetes.io/projected/5d14aa51-1b66-4d61-9515-61edbf06d2cb-kube-api-access-7pjdx\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.212610 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.212419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-tls\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.313586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.313586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjdx\" (UniqueName: \"kubernetes.io/projected/5d14aa51-1b66-4d61-9515-61edbf06d2cb-kube-api-access-7pjdx\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.313849 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-tls\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.313849 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.314016 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.314016 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.314016 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-grpc-tls\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.314016 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.313960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d14aa51-1b66-4d61-9515-61edbf06d2cb-metrics-client-ca\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.314869 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.314841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d14aa51-1b66-4d61-9515-61edbf06d2cb-metrics-client-ca\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.316583 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.316560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-grpc-tls\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.317183 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.317084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.317183 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.317122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.317340 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.317261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-tls\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.317599 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.317555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.319023 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.319003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d14aa51-1b66-4d61-9515-61edbf06d2cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.336627 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.336602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjdx\" (UniqueName: \"kubernetes.io/projected/5d14aa51-1b66-4d61-9515-61edbf06d2cb-kube-api-access-7pjdx\") pod \"thanos-querier-6b8b8448-gw5vz\" (UID: \"5d14aa51-1b66-4d61-9515-61edbf06d2cb\") " pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.425511 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.425474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:26.449783 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.449752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:26.525162 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.525100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:26.587426 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.587395 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b8b8448-gw5vz"] Jul 27 17:29:26.590471 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:29:26.590444 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d14aa51_1b66_4d61_9515_61edbf06d2cb.slice/crio-bb118c46c5a402a70d33587ab37cae2106ab06f96198db4c9bdf0fb3ee5257ac WatchSource:0}: Error finding container bb118c46c5a402a70d33587ab37cae2106ab06f96198db4c9bdf0fb3ee5257ac: Status 404 returned error can't find the container with id bb118c46c5a402a70d33587ab37cae2106ab06f96198db4c9bdf0fb3ee5257ac Jul 27 17:29:26.598349 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.598326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:26.862039 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.862001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" event={"ID":"5d14aa51-1b66-4d61-9515-61edbf06d2cb","Type":"ContainerStarted","Data":"bb118c46c5a402a70d33587ab37cae2106ab06f96198db4c9bdf0fb3ee5257ac"} Jul 27 17:29:26.864110 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.864080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mcrs9" event={"ID":"5c29a54b-c791-496a-8947-6e68af57a0e8","Type":"ContainerStarted","Data":"465ef11d82b7761050e4ea58ccd2afe566c4c09f18d4516ed2147b9390d7fb60"} Jul 27 17:29:26.864232 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.864132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mcrs9" event={"ID":"5c29a54b-c791-496a-8947-6e68af57a0e8","Type":"ContainerStarted","Data":"c777bc64b2cd104319e56b2b03207ab3e826ae321cd56f891c1e7f8ca7ecf46a"} Jul 27 17:29:26.894601 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:26.894558 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mcrs9" podStartSLOduration=2.201964503 podStartE2EDuration="3.894541327s" totalImagesPullingTime="1.630914647s" totalInitContainerRuntime="61.662177ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:26.892788778 +0000 UTC m=+82.891574610" watchObservedRunningTime="2026-07-27 17:29:26.894541327 +0000 UTC m=+82.893327148" Jul 27 17:29:27.640052 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:27.640018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:27.738851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:27.738819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:27.840024 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:27.839990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:29.356049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.356016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:29.446051 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.446005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:29.474036 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.469939 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:29:29.495209 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.495181 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:29:29.495369 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.495286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.498921 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.498880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 27 17:29:29.498921 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.498880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-csv66\"" Jul 27 17:29:29.499133 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.498887 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 27 17:29:29.499133 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.498905 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 27 17:29:29.499576 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.499522 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 27 17:29:29.500060 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.499698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 27 17:29:29.500060 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.499744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 27 17:29:29.500060 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.499831 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 27 17:29:29.500060 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.499925 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ll4t2h7rg2gl\"" Jul 27 17:29:29.500060 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.499967 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 27 17:29:29.500456 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.500253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 27 17:29:29.500456 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.500257 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 27 17:29:29.500569 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.500495 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 27 17:29:29.500960 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.500832 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 27 17:29:29.500960 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.500845 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 27 17:29:29.501170 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.501122 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 27 17:29:29.505025 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.504850 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 27 17:29:29.533408 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.533376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:29.643176 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643176 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643176 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643266 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-config-out\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2lg\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-kube-api-access-7s2lg\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-config\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-web-config\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643626 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.643980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.643731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.684701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.684646 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-674664878d-4pqch" podUID="772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" containerName="registry" containerID="cri-o://5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7" gracePeriod=30 Jul 27 17:29:29.744745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.744707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.744933 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.744766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.744933 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.744891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745050 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.744951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745050 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.744997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745050 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-config-out\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2lg\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-kube-api-access-7s2lg\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745438 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745438 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745438 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745438 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-config\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745438 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745438 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-web-config\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745791 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745791 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745791 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.745791 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.745504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.746241 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.746214 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.746724 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.746697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.747249 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.747220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.748508 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.748367 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.749178 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.748994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.750082 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.749569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.750082 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.749975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-web-config\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.751076 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.750551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.751076 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.751012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.751076 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.751034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.751517 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.751424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-config\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.751921 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.751879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.752072 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.752038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.752655 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.752607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.753820 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.753305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-config-out\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.753820 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.753801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.754120 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.754100 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.754410 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.754390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.757688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.757666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2lg\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-kube-api-access-7s2lg\") pod \"prometheus-k8s-0\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.776763 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.776717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:29.811407 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.811372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:29.886862 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:29.886811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:30.016477 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.016443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:30.055508 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.055484 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:29:30.098782 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.098751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:30.146834 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:29:30.146801 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c5d8aa_d171_4583_a06e_6853c9e93073.slice/crio-f3494bc0044beff5dd2dbeb91f660de1440cf6fbdf743aac333f7d92a7e3b49b WatchSource:0}: Error finding container f3494bc0044beff5dd2dbeb91f660de1440cf6fbdf743aac333f7d92a7e3b49b: Status 404 returned error can't find the container with id f3494bc0044beff5dd2dbeb91f660de1440cf6fbdf743aac333f7d92a7e3b49b Jul 27 17:29:30.164359 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.164341 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:29:30.248957 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.248933 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-ca-trust-extracted\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249052 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.248991 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-bound-sa-token\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249052 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249044 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfj9s\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-kube-api-access-sfj9s\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249076 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-certificates\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249228 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249188 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-installation-pull-secrets\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249282 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249247 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-image-registry-private-configuration\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249343 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249320 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-trusted-ca\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249395 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") pod \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\" (UID: \"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8\") " Jul 27 17:29:30.249774 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249747 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:29:30.249966 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.249943 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:29:30.251832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.251805 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:29:30.251926 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.251828 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:29:30.252118 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.252079 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:29:30.252290 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.252269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-kube-api-access-sfj9s" (OuterVolumeSpecName: "kube-api-access-sfj9s") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "kube-api-access-sfj9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:29:30.252341 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.252321 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:29:30.258491 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.258466 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" (UID: "772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:29:30.350742 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350719 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-trusted-ca\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.350742 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350743 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-tls\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.350995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350756 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-ca-trust-extracted\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.350995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350772 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-bound-sa-token\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.350995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350872 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfj9s\" (UniqueName: \"kubernetes.io/projected/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-kube-api-access-sfj9s\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.350995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350885 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-registry-certificates\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.350995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350901 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-installation-pull-secrets\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.350995 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.350916 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8-image-registry-private-configuration\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:29:30.884716 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.884677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" event={"ID":"5d14aa51-1b66-4d61-9515-61edbf06d2cb","Type":"ContainerStarted","Data":"84e71f771c9ccb2d444e0b6baaeeb1a55d4fd6d9914c3130aeba508500ce9057"} Jul 27 17:29:30.885160 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.884801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" event={"ID":"5d14aa51-1b66-4d61-9515-61edbf06d2cb","Type":"ContainerStarted","Data":"da5777cac61e5eedcb3be4679cfd7780c692b7d92d06dfb2e819ced8b1156428"} Jul 27 17:29:30.885160 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.884831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" event={"ID":"5d14aa51-1b66-4d61-9515-61edbf06d2cb","Type":"ContainerStarted","Data":"04e240ebf45871dab131949a0c0c73c103dd1795f0717d8588639974cd7afb54"} Jul 27 17:29:30.885987 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.885951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerStarted","Data":"f3494bc0044beff5dd2dbeb91f660de1440cf6fbdf743aac333f7d92a7e3b49b"} Jul 27 17:29:30.887175 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.887129 2564 generic.go:356] "Generic (PLEG): container finished" podID="772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" containerID="5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7" exitCode=0 Jul 27 17:29:30.887175 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.887167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674664878d-4pqch" event={"ID":"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8","Type":"ContainerDied","Data":"5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7"} Jul 27 17:29:30.887303 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.887195 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674664878d-4pqch" Jul 27 17:29:30.887303 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.887215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674664878d-4pqch" event={"ID":"772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8","Type":"ContainerDied","Data":"a22799895608f683317d15e07db3b7e6e90819495434d72befcf21dc0e70596a"} Jul 27 17:29:30.887303 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.887238 2564 scope.go:117] "RemoveContainer" containerID="5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7" Jul 27 17:29:30.895269 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.895244 2564 scope.go:117] "RemoveContainer" containerID="5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7" Jul 27 17:29:30.895573 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:29:30.895544 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7\": container with ID starting with 5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7 not found: ID does not exist" containerID="5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7" Jul 27 17:29:30.895638 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.895572 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7"} err="failed to get container status \"5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7\": rpc error: code = NotFound desc = could not find container \"5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7\": container with ID starting with 5698ffe56d05bf9b629a3310d0ecf27debad843ff87c130e4848afc2f235e4b7 not found: ID does not exist" Jul 27 17:29:30.911281 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.911258 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674664878d-4pqch"] Jul 27 17:29:30.919133 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:30.919109 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-674664878d-4pqch"] Jul 27 17:29:31.257892 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:31.257856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:31.324732 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:31.324702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:31.894628 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:31.894602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" event={"ID":"5d14aa51-1b66-4d61-9515-61edbf06d2cb","Type":"ContainerStarted","Data":"caf5651e3ce3b1abddfd78a5a4453513dbf27247e2a8ae42628463940b2973c9"} Jul 27 17:29:31.896342 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:31.896320 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" exitCode=0 Jul 27 17:29:31.896430 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:31.896355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc"} Jul 27 17:29:32.502498 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:32.502468 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" path="/var/lib/kubelet/pods/772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8/volumes" Jul 27 17:29:32.555753 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:32.555693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:32.622882 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:32.622854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:32.909426 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:32.909341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" event={"ID":"5d14aa51-1b66-4d61-9515-61edbf06d2cb","Type":"ContainerStarted","Data":"82e6e9eaca7efb4406614a39cc20c78e22b2cb5e76c501937324912b6c4d5349"} Jul 27 17:29:32.909850 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:32.909463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" event={"ID":"5d14aa51-1b66-4d61-9515-61edbf06d2cb","Type":"ContainerStarted","Data":"f0e011b7ebf62e4d28af1157ab8634cd22aa2ec73898eea86f30169f62e18b75"} Jul 27 17:29:32.909850 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:32.909630 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:32.933971 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:32.933924 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" podStartSLOduration=2.008805479 podStartE2EDuration="6.933906612s" totalImagesPullingTime="4.925101133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:32.932044717 +0000 UTC m=+88.930830541" watchObservedRunningTime="2026-07-27 17:29:32.933906612 +0000 UTC m=+88.932692435" Jul 27 17:29:35.446573 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.446542 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:35.542906 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.542865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:35.925707 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.925672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerStarted","Data":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} Jul 27 17:29:35.925870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.925762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerStarted","Data":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} Jul 27 17:29:35.925870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.925787 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerStarted","Data":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} Jul 27 17:29:35.925870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.925811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerStarted","Data":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} Jul 27 17:29:35.925870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.925834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerStarted","Data":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} Jul 27 17:29:35.925870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.925856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerStarted","Data":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} Jul 27 17:29:35.956608 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:35.956555 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.282570221 podStartE2EDuration="6.956540349s" totalImagesPullingTime="4.653534417s" totalInitContainerRuntime="20.435711ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:35.954809876 +0000 UTC m=+91.953595754" watchObservedRunningTime="2026-07-27 17:29:35.956540349 +0000 UTC m=+91.955326206" Jul 27 17:29:38.118020 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:38.117982 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:38.216062 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:38.216030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:38.920629 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:38.920603 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b8b8448-gw5vz" Jul 27 17:29:39.811727 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:39.811683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:29:40.690932 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:40.690901 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6458fc48fd-wzs5z"] Jul 27 17:29:40.799127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:40.799090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:40.859194 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:40.859128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:40.922032 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:40.921998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:46.267296 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:46.267255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:46.337592 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:46.337556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:46.821228 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:46.821196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w9rd7" Jul 27 17:29:54.143716 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:54.143683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:54.245683 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:54.245651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:29:54.302527 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:29:54.302496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:01.282064 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:01.282013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:01.385689 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:01.385656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:05.713834 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:05.713773 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6458fc48fd-wzs5z" podUID="e2d65e1e-eab2-4393-aea6-497182ca8d4f" containerName="console" containerID="cri-o://50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06" gracePeriod=15 Jul 27 17:30:05.857093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:05.857057 2564 patch_prober.go:28] interesting pod/console-6458fc48fd-wzs5z container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.12:8443/health\": dial tcp 10.134.0.12:8443: connect: connection refused" start-of-body= Jul 27 17:30:05.857255 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:05.857169 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6458fc48fd-wzs5z" podUID="e2d65e1e-eab2-4393-aea6-497182ca8d4f" containerName="console" probeResult="failure" output="Get \"https://10.134.0.12:8443/health\": dial tcp 10.134.0.12:8443: connect: connection refused" Jul 27 17:30:05.962102 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:05.962079 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:30:06.027603 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.027573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-service-ca\") pod \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " Jul 27 17:30:06.027755 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.027622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-trusted-ca-bundle\") pod \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " Jul 27 17:30:06.027755 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.027645 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-config\") pod \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " Jul 27 17:30:06.027755 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.027683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-oauth-serving-cert\") pod \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " Jul 27 17:30:06.027755 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.027700 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bzds\" (UniqueName: \"kubernetes.io/projected/e2d65e1e-eab2-4393-aea6-497182ca8d4f-kube-api-access-9bzds\") pod \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " Jul 27 17:30:06.027755 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.027721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-serving-cert\") pod \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " Jul 27 17:30:06.028006 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.027767 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-oauth-config\") pod \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\" (UID: \"e2d65e1e-eab2-4393-aea6-497182ca8d4f\") " Jul 27 17:30:06.028054 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.028012 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-service-ca" (OuterVolumeSpecName: "service-ca") pod "e2d65e1e-eab2-4393-aea6-497182ca8d4f" (UID: "e2d65e1e-eab2-4393-aea6-497182ca8d4f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:06.028184 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.028127 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-config" (OuterVolumeSpecName: "console-config") pod "e2d65e1e-eab2-4393-aea6-497182ca8d4f" (UID: "e2d65e1e-eab2-4393-aea6-497182ca8d4f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:06.028184 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.028176 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2d65e1e-eab2-4393-aea6-497182ca8d4f" (UID: "e2d65e1e-eab2-4393-aea6-497182ca8d4f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:06.028184 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.028193 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e2d65e1e-eab2-4393-aea6-497182ca8d4f" (UID: "e2d65e1e-eab2-4393-aea6-497182ca8d4f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:06.030034 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.030011 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e2d65e1e-eab2-4393-aea6-497182ca8d4f" (UID: "e2d65e1e-eab2-4393-aea6-497182ca8d4f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:06.030034 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.030014 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d65e1e-eab2-4393-aea6-497182ca8d4f-kube-api-access-9bzds" (OuterVolumeSpecName: "kube-api-access-9bzds") pod "e2d65e1e-eab2-4393-aea6-497182ca8d4f" (UID: "e2d65e1e-eab2-4393-aea6-497182ca8d4f"). InnerVolumeSpecName "kube-api-access-9bzds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:30:06.030198 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.030101 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e2d65e1e-eab2-4393-aea6-497182ca8d4f" (UID: "e2d65e1e-eab2-4393-aea6-497182ca8d4f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:06.040075 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.040053 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2d65e1e-eab2-4393-aea6-497182ca8d4f" containerID="50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06" exitCode=0 Jul 27 17:30:06.040192 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.040107 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6458fc48fd-wzs5z" Jul 27 17:30:06.040192 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.040120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6458fc48fd-wzs5z" event={"ID":"e2d65e1e-eab2-4393-aea6-497182ca8d4f","Type":"ContainerDied","Data":"50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06"} Jul 27 17:30:06.040192 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.040177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6458fc48fd-wzs5z" event={"ID":"e2d65e1e-eab2-4393-aea6-497182ca8d4f","Type":"ContainerDied","Data":"f909a0502690c6824834e6123aa780c5d34fb3c070de2aef1ba2761153909251"} Jul 27 17:30:06.040350 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.040197 2564 scope.go:117] "RemoveContainer" containerID="50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06" Jul 27 17:30:06.047826 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.047717 2564 scope.go:117] "RemoveContainer" containerID="50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06" Jul 27 17:30:06.048038 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:06.048006 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06\": container with ID starting with 50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06 not found: ID does not exist" containerID="50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06" Jul 27 17:30:06.048126 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.048039 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06"} err="failed to get container status \"50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06\": rpc error: code = NotFound desc = could not find container \"50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06\": container with ID starting with 50ac7f0a39b33556dfdf88667acb02bdb0d694314b5de971a8c2f520219efe06 not found: ID does not exist" Jul 27 17:30:06.060847 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.060827 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6458fc48fd-wzs5z"] Jul 27 17:30:06.070935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.070906 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6458fc48fd-wzs5z"] Jul 27 17:30:06.128993 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.128961 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-oauth-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:06.128993 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.128987 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzds\" (UniqueName: \"kubernetes.io/projected/e2d65e1e-eab2-4393-aea6-497182ca8d4f-kube-api-access-9bzds\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:06.128993 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.128997 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:06.129224 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.129005 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-oauth-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:06.129224 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.129015 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-service-ca\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:06.129224 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.129023 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-trusted-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:06.129224 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.129032 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2d65e1e-eab2-4393-aea6-497182ca8d4f-console-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:06.505232 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:06.505201 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d65e1e-eab2-4393-aea6-497182ca8d4f" path="/var/lib/kubelet/pods/e2d65e1e-eab2-4393-aea6-497182ca8d4f/volumes" Jul 27 17:30:13.612282 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:13.612251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:29.811898 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:29.811845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:29.832380 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:29.832353 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:30.141044 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:30.140967 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:31.504516 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:31.504483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:42.474308 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:42.474266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:42.907801 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:42.907707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:47.854257 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:47.854216 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:30:47.855251 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:47.855201 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-thanos" containerID="cri-o://78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" gracePeriod=600 Jul 27 17:30:47.855397 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:47.855344 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="config-reloader" containerID="cri-o://d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" gracePeriod=600 Jul 27 17:30:47.855647 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:47.855344 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-web" containerID="cri-o://4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" gracePeriod=600 Jul 27 17:30:47.855746 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:47.855388 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="thanos-sidecar" containerID="cri-o://7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" gracePeriod=600 Jul 27 17:30:47.856330 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:47.855396 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy" containerID="cri-o://ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" gracePeriod=600 Jul 27 17:30:47.856330 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:47.855123 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="prometheus" containerID="cri-o://7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" gracePeriod=600 Jul 27 17:30:48.100081 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.100045 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.158566 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158460 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-1\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158566 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158506 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-0\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158566 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158538 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-kube-rbac-proxy\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158599 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-tls-assets\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158629 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-web-config\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158695 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-grpc-tls\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.158851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-metrics-client-certs\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158864 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-kubelet-serving-ca-bundle\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158865 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158893 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-metrics-client-ca\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158922 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-config\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-trusted-ca-bundle\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.158986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-serving-certs-ca-bundle\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159023 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-tls\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-db\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159093 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-config-out\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159128 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-thanos-prometheus-http-client-file\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159166 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159165 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s2lg\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-kube-api-access-7s2lg\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-2\") pod \"83c5d8aa-d171-4583-a06e-6853c9e93073\" (UID: \"83c5d8aa-d171-4583-a06e-6853c9e93073\") " Jul 27 17:30:48.159745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159244 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:48.159745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159426 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.159745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159446 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.159745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159612 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:48.159745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159678 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:48.160041 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.159943 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:48.161501 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.161467 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.161803 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.161778 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:48.162194 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.162171 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:48.162281 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.162238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.165314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.162782 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:30:48.165314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.163407 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:30:48.165314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.163683 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.165314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.164640 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.165314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.164840 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.165314 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.165203 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.165678 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.165483 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-config" (OuterVolumeSpecName: "config") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.167042 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.166219 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.167042 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.166856 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-kube-api-access-7s2lg" (OuterVolumeSpecName: "kube-api-access-7s2lg") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "kube-api-access-7s2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:30:48.167237 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.167200 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-config-out" (OuterVolumeSpecName: "config-out") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:30:48.175810 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.175778 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-web-config" (OuterVolumeSpecName: "web-config") pod "83c5d8aa-d171-4583-a06e-6853c9e93073" (UID: "83c5d8aa-d171-4583-a06e-6853c9e93073"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:48.191734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191708 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" exitCode=0 Jul 27 17:30:48.191734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191729 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" exitCode=0 Jul 27 17:30:48.191734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191736 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" exitCode=0 Jul 27 17:30:48.191895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191742 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" exitCode=0 Jul 27 17:30:48.191895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191748 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" exitCode=0 Jul 27 17:30:48.191895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191755 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" exitCode=0 Jul 27 17:30:48.191895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191781 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.191895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} Jul 27 17:30:48.191895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} Jul 27 17:30:48.191895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} Jul 27 17:30:48.192186 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} Jul 27 17:30:48.192186 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191930 2564 scope.go:117] "RemoveContainer" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.192186 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.191933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} Jul 27 17:30:48.192186 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.192082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} Jul 27 17:30:48.192186 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.192130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83c5d8aa-d171-4583-a06e-6853c9e93073","Type":"ContainerDied","Data":"f3494bc0044beff5dd2dbeb91f660de1440cf6fbdf743aac333f7d92a7e3b49b"} Jul 27 17:30:48.200422 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.200359 2564 scope.go:117] "RemoveContainer" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.207167 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.207131 2564 scope.go:117] "RemoveContainer" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.213534 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.213517 2564 scope.go:117] "RemoveContainer" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.217737 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.217714 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:30:48.221216 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.221197 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:30:48.222708 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.222686 2564 scope.go:117] "RemoveContainer" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.229243 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.229228 2564 scope.go:117] "RemoveContainer" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.235930 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.235915 2564 scope.go:117] "RemoveContainer" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.242391 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.242375 2564 scope.go:117] "RemoveContainer" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.242658 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:48.242639 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": container with ID starting with 78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e not found: ID does not exist" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.242731 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.242669 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} err="failed to get container status \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": rpc error: code = NotFound desc = could not find container \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": container with ID starting with 78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e not found: ID does not exist" Jul 27 17:30:48.242731 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.242698 2564 scope.go:117] "RemoveContainer" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.242931 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:48.242911 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": container with ID starting with ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c not found: ID does not exist" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.242969 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.242938 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} err="failed to get container status \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": rpc error: code = NotFound desc = could not find container \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": container with ID starting with ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c not found: ID does not exist" Jul 27 17:30:48.242969 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.242956 2564 scope.go:117] "RemoveContainer" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.243180 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:48.243159 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": container with ID starting with 4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb not found: ID does not exist" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.243250 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243187 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} err="failed to get container status \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": rpc error: code = NotFound desc = could not find container \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": container with ID starting with 4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb not found: ID does not exist" Jul 27 17:30:48.243250 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243209 2564 scope.go:117] "RemoveContainer" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.243424 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:48.243407 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": container with ID starting with 7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424 not found: ID does not exist" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.243483 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243430 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} err="failed to get container status \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": rpc error: code = NotFound desc = could not find container \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": container with ID starting with 7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424 not found: ID does not exist" Jul 27 17:30:48.243483 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243449 2564 scope.go:117] "RemoveContainer" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.243696 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:48.243681 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": container with ID starting with d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd not found: ID does not exist" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.243760 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243701 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} err="failed to get container status \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": rpc error: code = NotFound desc = could not find container \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": container with ID starting with d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd not found: ID does not exist" Jul 27 17:30:48.243760 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243719 2564 scope.go:117] "RemoveContainer" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.243923 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:48.243905 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": container with ID starting with 7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d not found: ID does not exist" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.243958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243927 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} err="failed to get container status \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": rpc error: code = NotFound desc = could not find container \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": container with ID starting with 7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d not found: ID does not exist" Jul 27 17:30:48.243958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.243941 2564 scope.go:117] "RemoveContainer" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.244174 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:30:48.244129 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": container with ID starting with ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc not found: ID does not exist" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.244220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244180 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc"} err="failed to get container status \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": rpc error: code = NotFound desc = could not find container \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": container with ID starting with ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc not found: ID does not exist" Jul 27 17:30:48.244220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244192 2564 scope.go:117] "RemoveContainer" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.244391 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244374 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} err="failed to get container status \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": rpc error: code = NotFound desc = could not find container \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": container with ID starting with 78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e not found: ID does not exist" Jul 27 17:30:48.244391 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244393 2564 scope.go:117] "RemoveContainer" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.244597 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244579 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} err="failed to get container status \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": rpc error: code = NotFound desc = could not find container \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": container with ID starting with ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c not found: ID does not exist" Jul 27 17:30:48.244643 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244598 2564 scope.go:117] "RemoveContainer" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.244808 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244789 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} err="failed to get container status \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": rpc error: code = NotFound desc = could not find container \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": container with ID starting with 4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb not found: ID does not exist" Jul 27 17:30:48.244848 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244811 2564 scope.go:117] "RemoveContainer" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.245006 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.244990 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} err="failed to get container status \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": rpc error: code = NotFound desc = could not find container \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": container with ID starting with 7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424 not found: ID does not exist" Jul 27 17:30:48.245006 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245005 2564 scope.go:117] "RemoveContainer" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.245275 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245260 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} err="failed to get container status \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": rpc error: code = NotFound desc = could not find container \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": container with ID starting with d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd not found: ID does not exist" Jul 27 17:30:48.245275 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245275 2564 scope.go:117] "RemoveContainer" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.245473 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245456 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} err="failed to get container status \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": rpc error: code = NotFound desc = could not find container \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": container with ID starting with 7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d not found: ID does not exist" Jul 27 17:30:48.245520 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245474 2564 scope.go:117] "RemoveContainer" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.245666 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245648 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc"} err="failed to get container status \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": rpc error: code = NotFound desc = could not find container \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": container with ID starting with ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc not found: ID does not exist" Jul 27 17:30:48.245666 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245665 2564 scope.go:117] "RemoveContainer" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.245843 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245827 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} err="failed to get container status \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": rpc error: code = NotFound desc = could not find container \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": container with ID starting with 78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e not found: ID does not exist" Jul 27 17:30:48.245884 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245842 2564 scope.go:117] "RemoveContainer" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.246009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.245994 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} err="failed to get container status \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": rpc error: code = NotFound desc = could not find container \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": container with ID starting with ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c not found: ID does not exist" Jul 27 17:30:48.246049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.246009 2564 scope.go:117] "RemoveContainer" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.246188 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.246172 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} err="failed to get container status \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": rpc error: code = NotFound desc = could not find container \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": container with ID starting with 4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb not found: ID does not exist" Jul 27 17:30:48.246237 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.246188 2564 scope.go:117] "RemoveContainer" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.246447 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.246419 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} err="failed to get container status \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": rpc error: code = NotFound desc = could not find container \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": container with ID starting with 7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424 not found: ID does not exist" Jul 27 17:30:48.246447 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.246443 2564 scope.go:117] "RemoveContainer" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.246742 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.246716 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} err="failed to get container status \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": rpc error: code = NotFound desc = could not find container \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": container with ID starting with d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd not found: ID does not exist" Jul 27 17:30:48.247022 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.246741 2564 scope.go:117] "RemoveContainer" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.247360 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.247340 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} err="failed to get container status \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": rpc error: code = NotFound desc = could not find container \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": container with ID starting with 7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d not found: ID does not exist" Jul 27 17:30:48.247458 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.247370 2564 scope.go:117] "RemoveContainer" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.247638 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.247620 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc"} err="failed to get container status \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": rpc error: code = NotFound desc = could not find container \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": container with ID starting with ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc not found: ID does not exist" Jul 27 17:30:48.247693 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.247639 2564 scope.go:117] "RemoveContainer" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.247870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.247855 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} err="failed to get container status \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": rpc error: code = NotFound desc = could not find container \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": container with ID starting with 78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e not found: ID does not exist" Jul 27 17:30:48.247914 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.247871 2564 scope.go:117] "RemoveContainer" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.248100 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248081 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} err="failed to get container status \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": rpc error: code = NotFound desc = could not find container \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": container with ID starting with ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c not found: ID does not exist" Jul 27 17:30:48.248188 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248102 2564 scope.go:117] "RemoveContainer" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.248271 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248248 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:30:48.248434 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248417 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} err="failed to get container status \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": rpc error: code = NotFound desc = could not find container \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": container with ID starting with 4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb not found: ID does not exist" Jul 27 17:30:48.248485 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248435 2564 scope.go:117] "RemoveContainer" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.248545 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248531 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="prometheus" Jul 27 17:30:48.248589 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248549 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="prometheus" Jul 27 17:30:48.248589 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248566 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-web" Jul 27 17:30:48.248589 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248576 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-web" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248591 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="config-reloader" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248597 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="config-reloader" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248607 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248615 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248622 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="init-config-reloader" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248628 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="init-config-reloader" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248638 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="thanos-sidecar" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248646 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="thanos-sidecar" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248659 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" containerName="registry" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248664 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" containerName="registry" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248657 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} err="failed to get container status \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": rpc error: code = NotFound desc = could not find container \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": container with ID starting with 7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424 not found: ID does not exist" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248674 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d65e1e-eab2-4393-aea6-497182ca8d4f" containerName="console" Jul 27 17:30:48.248688 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248677 2564 scope.go:117] "RemoveContainer" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248679 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d65e1e-eab2-4393-aea6-497182ca8d4f" containerName="console" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248731 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-thanos" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248739 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-thanos" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248800 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="config-reloader" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248809 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248819 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="772f3d3c-60c2-4e11-8a26-ee8ae03ec5a8" containerName="registry" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248831 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="prometheus" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248837 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="thanos-sidecar" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248843 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-thanos" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248851 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d65e1e-eab2-4393-aea6-497182ca8d4f" containerName="console" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248860 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" containerName="kube-rbac-proxy-web" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248925 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} err="failed to get container status \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": rpc error: code = NotFound desc = could not find container \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": container with ID starting with d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd not found: ID does not exist" Jul 27 17:30:48.249199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.248949 2564 scope.go:117] "RemoveContainer" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.249623 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249213 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} err="failed to get container status \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": rpc error: code = NotFound desc = could not find container \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": container with ID starting with 7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d not found: ID does not exist" Jul 27 17:30:48.249623 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249229 2564 scope.go:117] "RemoveContainer" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.249623 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249440 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc"} err="failed to get container status \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": rpc error: code = NotFound desc = could not find container \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": container with ID starting with ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc not found: ID does not exist" Jul 27 17:30:48.249623 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249456 2564 scope.go:117] "RemoveContainer" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.249760 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249672 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} err="failed to get container status \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": rpc error: code = NotFound desc = could not find container \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": container with ID starting with 78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e not found: ID does not exist" Jul 27 17:30:48.249760 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249686 2564 scope.go:117] "RemoveContainer" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.249914 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249891 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} err="failed to get container status \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": rpc error: code = NotFound desc = could not find container \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": container with ID starting with ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c not found: ID does not exist" Jul 27 17:30:48.249956 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.249917 2564 scope.go:117] "RemoveContainer" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.250137 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.250120 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} err="failed to get container status \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": rpc error: code = NotFound desc = could not find container \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": container with ID starting with 4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb not found: ID does not exist" Jul 27 17:30:48.250206 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.250172 2564 scope.go:117] "RemoveContainer" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.250489 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.250463 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} err="failed to get container status \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": rpc error: code = NotFound desc = could not find container \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": container with ID starting with 7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424 not found: ID does not exist" Jul 27 17:30:48.250583 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.250499 2564 scope.go:117] "RemoveContainer" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.256480 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.254937 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.257220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.257055 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} err="failed to get container status \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": rpc error: code = NotFound desc = could not find container \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": container with ID starting with d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd not found: ID does not exist" Jul 27 17:30:48.257220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.257101 2564 scope.go:117] "RemoveContainer" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.259079 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.258854 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} err="failed to get container status \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": rpc error: code = NotFound desc = could not find container \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": container with ID starting with 7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d not found: ID does not exist" Jul 27 17:30:48.259079 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.258872 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-csv66\"" Jul 27 17:30:48.259079 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.258911 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 27 17:30:48.259079 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.258884 2564 scope.go:117] "RemoveContainer" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.259079 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.258971 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 27 17:30:48.259512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259271 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc"} err="failed to get container status \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": rpc error: code = NotFound desc = could not find container \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": container with ID starting with ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc not found: ID does not exist" Jul 27 17:30:48.259512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259296 2564 scope.go:117] "RemoveContainer" containerID="78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e" Jul 27 17:30:48.259512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259369 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ll4t2h7rg2gl\"" Jul 27 17:30:48.259512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259409 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 27 17:30:48.259512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259434 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 27 17:30:48.259512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259436 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 27 17:30:48.259958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259588 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 27 17:30:48.259958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.259958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.259958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.259958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.259978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260003 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260006 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-config\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260103 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e"} err="failed to get container status \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": rpc error: code = NotFound desc = could not find container \"78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e\": container with ID starting with 78d72e0950f0f759c585aeed5759541363ed1b4c40098c240db6c156fc69d05e not found: ID does not exist" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260129 2564 scope.go:117] "RemoveContainer" containerID="ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c" Jul 27 17:30:48.260214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0bd78e95-4c46-4239-a83b-80112fabfbc4-config-out\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260302 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-web-config\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260301 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0bd78e95-4c46-4239-a83b-80112fabfbc4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260632 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnpz\" (UniqueName: \"kubernetes.io/projected/0bd78e95-4c46-4239-a83b-80112fabfbc4-kube-api-access-glnpz\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.260880 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.260986 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c"} err="failed to get container status \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": rpc error: code = NotFound desc = could not find container \"ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c\": container with ID starting with ff841d8143865703e5d4e2dfc445ef49aac3d9b694b0373fbb1763c95167b39c not found: ID does not exist" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261017 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-db\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261045 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83c5d8aa-d171-4583-a06e-6853c9e93073-config-out\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261074 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261025 2564 scope.go:117] "RemoveContainer" containerID="4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261099 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2lg\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-kube-api-access-7s2lg\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261200 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261219 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261254 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-kube-rbac-proxy\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261276 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261291 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83c5d8aa-d171-4583-a06e-6853c9e93073-tls-assets\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261310 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-web-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261328 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-grpc-tls\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261346 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261364 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-metrics-client-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261390 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-metrics-client-ca\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261403 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261401 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb"} err="failed to get container status \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": rpc error: code = NotFound desc = could not find container \"4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb\": container with ID starting with 4dbedd1c37118060942bd8366b118ee836633e9713303e198526521ad4ebb4cb not found: ID does not exist" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261446 2564 scope.go:117] "RemoveContainer" containerID="7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424" Jul 27 17:30:48.261734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261422 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261643 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83c5d8aa-d171-4583-a06e-6853c9e93073-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261662 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83c5d8aa-d171-4583-a06e-6853c9e93073-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261740 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424"} err="failed to get container status \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": rpc error: code = NotFound desc = could not find container \"7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424\": container with ID starting with 7ee878e7d19276b32e249def7829a6f01d77bd07ed5e127971d6df5f1b243424 not found: ID does not exist" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261763 2564 scope.go:117] "RemoveContainer" containerID="d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261820 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.261989 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd"} err="failed to get container status \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": rpc error: code = NotFound desc = could not find container \"d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd\": container with ID starting with d217ea0d15578e289dca90b3d3e5e1ea7cfe3d27f5f3c2cd5a36e8e597f495cd not found: ID does not exist" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.262010 2564 scope.go:117] "RemoveContainer" containerID="7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.262360 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d"} err="failed to get container status \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": rpc error: code = NotFound desc = could not find container \"7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d\": container with ID starting with 7224714ed5a6ee4712a2db7127dd343c6829781f5d8d347db43d8c7a5268a90d not found: ID does not exist" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.262385 2564 scope.go:117] "RemoveContainer" containerID="ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc" Jul 27 17:30:48.262832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.262658 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc"} err="failed to get container status \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": rpc error: code = NotFound desc = could not find container \"ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc\": container with ID starting with ff4415d2dc0970309ac46b36f626eca56ff103a9a4324b4bd3dd1662ab9a29fc not found: ID does not exist" Jul 27 17:30:48.263586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.263567 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 27 17:30:48.265457 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.265438 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 27 17:30:48.267947 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.267929 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:30:48.362716 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362716 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0bd78e95-4c46-4239-a83b-80112fabfbc4-config-out\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-web-config\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362810 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0bd78e95-4c46-4239-a83b-80112fabfbc4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.362935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnpz\" (UniqueName: \"kubernetes.io/projected/0bd78e95-4c46-4239-a83b-80112fabfbc4-kube-api-access-glnpz\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.362964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-config\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363837 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363566 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.363837 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.363569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.365666 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.364807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.365666 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.364974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.365666 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.365046 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.365666 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.365093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.365666 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.365623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.366049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.365883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0bd78e95-4c46-4239-a83b-80112fabfbc4-config-out\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.366049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.365957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.367701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.366205 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.367701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.366401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-web-config\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.367701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.367096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.367701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.367652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.367701 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.367669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0bd78e95-4c46-4239-a83b-80112fabfbc4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.368241 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.368221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.368293 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.368256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.368431 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.368408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.368472 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.368439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0bd78e95-4c46-4239-a83b-80112fabfbc4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.368910 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.368894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0bd78e95-4c46-4239-a83b-80112fabfbc4-config\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.372043 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.372022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnpz\" (UniqueName: \"kubernetes.io/projected/0bd78e95-4c46-4239-a83b-80112fabfbc4-kube-api-access-glnpz\") pod \"prometheus-k8s-0\" (UID: \"0bd78e95-4c46-4239-a83b-80112fabfbc4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.502925 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.502881 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c5d8aa-d171-4583-a06e-6853c9e93073" path="/var/lib/kubelet/pods/83c5d8aa-d171-4583-a06e-6853c9e93073/volumes" Jul 27 17:30:48.566778 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.566743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:48.701385 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:48.701352 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 27 17:30:48.704360 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:30:48.704333 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd78e95_4c46_4239_a83b_80112fabfbc4.slice/crio-855d625a91ef226ec1f22c8f88c30985cc28c11fcb80cec2aab9b44220bb45cf WatchSource:0}: Error finding container 855d625a91ef226ec1f22c8f88c30985cc28c11fcb80cec2aab9b44220bb45cf: Status 404 returned error can't find the container with id 855d625a91ef226ec1f22c8f88c30985cc28c11fcb80cec2aab9b44220bb45cf Jul 27 17:30:49.197863 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:49.197773 2564 generic.go:356] "Generic (PLEG): container finished" podID="0bd78e95-4c46-4239-a83b-80112fabfbc4" containerID="43ddedc032db5add6b835c184345137c4b7adf0a1c61744d6d29a34b0d228481" exitCode=0 Jul 27 17:30:49.197863 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:49.197845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerDied","Data":"43ddedc032db5add6b835c184345137c4b7adf0a1c61744d6d29a34b0d228481"} Jul 27 17:30:49.198391 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:49.197916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerStarted","Data":"855d625a91ef226ec1f22c8f88c30985cc28c11fcb80cec2aab9b44220bb45cf"} Jul 27 17:30:50.204578 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:50.204543 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerStarted","Data":"8d7973872c5a4053d820c8146bd6166cdfc732527e3cb93c3c8a34e34ec4fe1d"} Jul 27 17:30:50.205049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:50.204659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerStarted","Data":"840e0cbd14aa054ed4be76f2c9f156e92d704ac5dd041f8107c097e560d3b364"} Jul 27 17:30:50.205049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:50.204707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerStarted","Data":"d7789df846ad2848a97d1ae26ed0e152bb9e00f6c3eac9016654fb514d32fd7d"} Jul 27 17:30:50.205049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:50.204756 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerStarted","Data":"b312067a7eea49b735bf4c1287dde40b7fb0789432b68639b592c32edeb86b94"} Jul 27 17:30:50.205049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:50.204793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerStarted","Data":"49d3dd60baa7aab3ffff8204a2033124d4eb24e5ff772d8d12d9d703fbb0aa89"} Jul 27 17:30:50.205049 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:50.204832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0bd78e95-4c46-4239-a83b-80112fabfbc4","Type":"ContainerStarted","Data":"01885ddbb432ab8c608fa35839ec3a323f080c576aa40cfa6601ece3a8f723f5"} Jul 27 17:30:50.235333 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:50.235053 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.217972483 podStartE2EDuration="2.235034441s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.061958ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:50.232916133 +0000 UTC m=+166.231701980" watchObservedRunningTime="2026-07-27 17:30:50.235034441 +0000 UTC m=+166.233820262" Jul 27 17:30:53.566960 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:53.566918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:30:54.050265 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:54.050229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:54.150455 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:54.150393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:54.701515 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:54.701476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:30:54.754298 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:30:54.754269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:05.149930 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:05.149894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:05.249519 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:05.249478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:05.303375 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:05.303340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:42.645429 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:42.645396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:48.567829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:48.567788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:31:48.582915 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:48.582889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:31:49.435434 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:49.435407 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 27 17:31:57.739539 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:57.739496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:31:57.830270 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:31:57.830230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:01.373928 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:01.373874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:01.430656 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:01.430622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:09.671835 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:09.671791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:09.731771 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:09.731739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:12.394698 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:12.394659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:12.495122 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:12.495090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:12.596556 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:12.596525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:15.677779 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:15.677744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:16.267852 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:16.267810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:16.368909 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:16.368870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:19.143562 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:19.143531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:19.239876 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:19.239844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:51.791206 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:51.791169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:51.891280 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:51.891238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:59.754209 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:59.754136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:32:59.815387 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:32:59.815347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:00.851787 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:00.851741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:01.003756 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:01.003705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:01.067280 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:01.067248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:04.355193 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:04.355166 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 17:33:50.256586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:50.256528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:52.398473 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:52.398419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:52.499173 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:52.499125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:52.599734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:52.599702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:33:52.700089 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:33:52.700003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:15.044275 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:15.044192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:15.142234 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:15.142198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:20.566441 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.566405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:20.860103 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.860015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:20.864745 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.864718 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-ln9dd"] Jul 27 17:34:20.867719 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.867703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:20.870521 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.870499 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 27 17:34:20.870647 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.870536 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 27 17:34:20.870647 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.870572 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 27 17:34:20.870647 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.870586 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 27 17:34:20.870647 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.870625 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 27 17:34:20.872047 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.872028 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-2r9g4\"" Jul 27 17:34:20.877493 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.877473 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-ln9dd"] Jul 27 17:34:20.893859 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.893831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/de3d765c-369b-4693-b460-37ea386c9af0-cabundle0\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:20.893994 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.893872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-certificates\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:20.893994 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.893908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dw47\" (UniqueName: \"kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-kube-api-access-5dw47\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:20.970076 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.970044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:20.995269 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.995228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dw47\" (UniqueName: \"kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-kube-api-access-5dw47\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:20.995395 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.995311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/de3d765c-369b-4693-b460-37ea386c9af0-cabundle0\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:20.995395 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.995336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-certificates\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:20.995474 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:34:20.995425 2564 secret.go:281] references non-existent secret key: ca.crt Jul 27 17:34:20.995474 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:34:20.995437 2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 27 17:34:20.995474 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:34:20.995444 2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-ln9dd: references non-existent secret key: ca.crt Jul 27 17:34:20.995579 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:34:20.995499 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-certificates podName:de3d765c-369b-4693-b460-37ea386c9af0 nodeName:}" failed. No retries permitted until 2026-07-27 17:34:21.495483003 +0000 UTC m=+377.494268802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-certificates") pod "keda-operator-77f648d694-ln9dd" (UID: "de3d765c-369b-4693-b460-37ea386c9af0") : references non-existent secret key: ca.crt Jul 27 17:34:20.995940 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:20.995922 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/de3d765c-369b-4693-b460-37ea386c9af0-cabundle0\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:21.003868 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.003840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dw47\" (UniqueName: \"kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-kube-api-access-5dw47\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:21.062906 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.062868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:21.133940 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.133846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:21.329960 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.329925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:21.431471 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.431384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:21.499512 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.499466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-certificates\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:21.501957 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.501931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de3d765c-369b-4693-b460-37ea386c9af0-certificates\") pod \"keda-operator-77f648d694-ln9dd\" (UID: \"de3d765c-369b-4693-b460-37ea386c9af0\") " pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:21.503659 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.503628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:21.778222 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.778188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:21.837231 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.837164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:21.940543 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.940467 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-ln9dd"] Jul 27 17:34:21.946585 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:34:21.946552 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3d765c_369b_4693_b460_37ea386c9af0.slice/crio-db6c90d4306ee8c571c898c4175389617026eb379fe59754745d83a0ff827e9f WatchSource:0}: Error finding container db6c90d4306ee8c571c898c4175389617026eb379fe59754745d83a0ff827e9f: Status 404 returned error can't find the container with id db6c90d4306ee8c571c898c4175389617026eb379fe59754745d83a0ff827e9f Jul 27 17:34:21.947834 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.947817 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:34:21.988302 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:21.988266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:22.188308 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:22.188230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:22.248295 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:22.248260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:34:22.949019 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:22.948975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-ln9dd" event={"ID":"de3d765c-369b-4693-b460-37ea386c9af0","Type":"ContainerStarted","Data":"db6c90d4306ee8c571c898c4175389617026eb379fe59754745d83a0ff827e9f"} Jul 27 17:34:25.963825 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:25.963784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-ln9dd" event={"ID":"de3d765c-369b-4693-b460-37ea386c9af0","Type":"ContainerStarted","Data":"b327d17be616d0b508c2724fdffb41acde5927443135b144d7f08bf9c79f5aa0"} Jul 27 17:34:25.964324 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:25.963928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:34:25.981052 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:25.981005 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-ln9dd" podStartSLOduration=2.603654368 podStartE2EDuration="5.980991264s" totalImagesPullingTime="3.377336896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:34:25.98040867 +0000 UTC m=+381.979194491" watchObservedRunningTime="2026-07-27 17:34:25.980991264 +0000 UTC m=+381.979777084" Jul 27 17:34:46.969505 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:34:46.969473 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-ln9dd" Jul 27 17:35:37.708632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.708547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:37.772579 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.772545 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bsbsw"] Jul 27 17:35:37.776236 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.776212 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:37.776488 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.776462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:37.779126 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.779106 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-z8nx7\"" Jul 27 17:35:37.779276 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.779180 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 27 17:35:37.780304 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.780284 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 27 17:35:37.780431 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.780318 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 27 17:35:37.788669 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.788649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bsbsw"] Jul 27 17:35:37.811545 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.811518 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-g2kfm"] Jul 27 17:35:37.814528 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.814512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:37.817484 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.817463 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-t2xht\"" Jul 27 17:35:37.817587 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.817502 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 27 17:35:37.826348 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.826327 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-g2kfm"] Jul 27 17:35:37.868884 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.868861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:37.897168 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.897128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbd8\" (UniqueName: \"kubernetes.io/projected/5d2bf4df-74a8-4311-8077-ac8e652878bd-kube-api-access-8bbd8\") pod \"kserve-controller-manager-ff96d5959-bsbsw\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:37.897255 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.897193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1da230c8-be4b-4648-8138-ec6e80d1fbf1-data\") pod \"seaweedfs-64568bcd49-g2kfm\" (UID: \"1da230c8-be4b-4648-8138-ec6e80d1fbf1\") " pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:37.897301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.897272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2bf4df-74a8-4311-8077-ac8e652878bd-cert\") pod \"kserve-controller-manager-ff96d5959-bsbsw\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:37.897340 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.897309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkmf\" (UniqueName: \"kubernetes.io/projected/1da230c8-be4b-4648-8138-ec6e80d1fbf1-kube-api-access-wxkmf\") pod \"seaweedfs-64568bcd49-g2kfm\" (UID: \"1da230c8-be4b-4648-8138-ec6e80d1fbf1\") " pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:37.971327 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.971264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:37.997845 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.997821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2bf4df-74a8-4311-8077-ac8e652878bd-cert\") pod \"kserve-controller-manager-ff96d5959-bsbsw\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:37.997912 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.997855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkmf\" (UniqueName: \"kubernetes.io/projected/1da230c8-be4b-4648-8138-ec6e80d1fbf1-kube-api-access-wxkmf\") pod \"seaweedfs-64568bcd49-g2kfm\" (UID: \"1da230c8-be4b-4648-8138-ec6e80d1fbf1\") " pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:37.997912 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.997887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbd8\" (UniqueName: \"kubernetes.io/projected/5d2bf4df-74a8-4311-8077-ac8e652878bd-kube-api-access-8bbd8\") pod \"kserve-controller-manager-ff96d5959-bsbsw\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:37.997978 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.997934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1da230c8-be4b-4648-8138-ec6e80d1fbf1-data\") pod \"seaweedfs-64568bcd49-g2kfm\" (UID: \"1da230c8-be4b-4648-8138-ec6e80d1fbf1\") " pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:37.998367 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:37.998342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1da230c8-be4b-4648-8138-ec6e80d1fbf1-data\") pod \"seaweedfs-64568bcd49-g2kfm\" (UID: \"1da230c8-be4b-4648-8138-ec6e80d1fbf1\") " pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:38.000298 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.000272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2bf4df-74a8-4311-8077-ac8e652878bd-cert\") pod \"kserve-controller-manager-ff96d5959-bsbsw\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:38.007938 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.007918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkmf\" (UniqueName: \"kubernetes.io/projected/1da230c8-be4b-4648-8138-ec6e80d1fbf1-kube-api-access-wxkmf\") pod \"seaweedfs-64568bcd49-g2kfm\" (UID: \"1da230c8-be4b-4648-8138-ec6e80d1fbf1\") " pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:38.008320 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.008304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbd8\" (UniqueName: \"kubernetes.io/projected/5d2bf4df-74a8-4311-8077-ac8e652878bd-kube-api-access-8bbd8\") pod \"kserve-controller-manager-ff96d5959-bsbsw\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:38.088733 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.088706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:38.126508 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.126478 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:38.225657 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.225630 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bsbsw"] Jul 27 17:35:38.227987 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:35:38.227950 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2bf4df_74a8_4311_8077_ac8e652878bd.slice/crio-5170983beb7dbfb555f7ad3de9ad61875ee4db7056cee4730e0459a3ab6c1247 WatchSource:0}: Error finding container 5170983beb7dbfb555f7ad3de9ad61875ee4db7056cee4730e0459a3ab6c1247: Status 404 returned error can't find the container with id 5170983beb7dbfb555f7ad3de9ad61875ee4db7056cee4730e0459a3ab6c1247 Jul 27 17:35:38.278029 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.278007 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-g2kfm"] Jul 27 17:35:38.279770 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:35:38.279740 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da230c8_be4b_4648_8138_ec6e80d1fbf1.slice/crio-5dbeb1642a512022a35b1a0f0d52da77c5bf5ba89b39179e4be21d8846369078 WatchSource:0}: Error finding container 5dbeb1642a512022a35b1a0f0d52da77c5bf5ba89b39179e4be21d8846369078: Status 404 returned error can't find the container with id 5dbeb1642a512022a35b1a0f0d52da77c5bf5ba89b39179e4be21d8846369078 Jul 27 17:35:38.385737 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.385705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:38.483857 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.483784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:38.584220 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:38.584192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:39.226135 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:39.226096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-g2kfm" event={"ID":"1da230c8-be4b-4648-8138-ec6e80d1fbf1","Type":"ContainerStarted","Data":"5dbeb1642a512022a35b1a0f0d52da77c5bf5ba89b39179e4be21d8846369078"} Jul 27 17:35:39.227247 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:39.227216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" event={"ID":"5d2bf4df-74a8-4311-8077-ac8e652878bd","Type":"ContainerStarted","Data":"5170983beb7dbfb555f7ad3de9ad61875ee4db7056cee4730e0459a3ab6c1247"} Jul 27 17:35:43.246185 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:43.246079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" event={"ID":"5d2bf4df-74a8-4311-8077-ac8e652878bd","Type":"ContainerStarted","Data":"ca341b246388efe3672f5ecafbb8fd847a477d5ca88a1666648ce1783beb5be4"} Jul 27 17:35:43.246185 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:43.246182 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:35:43.247730 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:43.247708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-g2kfm" event={"ID":"1da230c8-be4b-4648-8138-ec6e80d1fbf1","Type":"ContainerStarted","Data":"c3e4f5818bf8064c5f9402b722280c0deb91b928a7987aca960c037cc2e27038"} Jul 27 17:35:43.247866 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:43.247822 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:43.264542 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:43.264498 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" podStartSLOduration=1.617040829 podStartE2EDuration="6.264483283s" totalImagesPullingTime="4.647442454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:43.262492485 +0000 UTC m=+459.261278306" watchObservedRunningTime="2026-07-27 17:35:43.264483283 +0000 UTC m=+459.263269105" Jul 27 17:35:43.278948 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:43.278907 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-g2kfm" podStartSLOduration=1.628030198 podStartE2EDuration="6.278892634s" totalImagesPullingTime="4.650862436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:43.278290034 +0000 UTC m=+459.277075867" watchObservedRunningTime="2026-07-27 17:35:43.278892634 +0000 UTC m=+459.277678452" Jul 27 17:35:49.253672 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:49.253640 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-g2kfm" Jul 27 17:35:52.188213 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:52.188173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:52.256634 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:52.256602 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:54.015705 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:54.015670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:35:54.071664 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:35:54.071629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:01.990451 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.990415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c95477bc9-l7fk8"] Jul 27 17:36:01.993626 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.993608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:01.998586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 17:36:01.998586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998452 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 17:36:01.998586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998459 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c76kr\"" Jul 27 17:36:01.998586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998509 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 17:36:01.998586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998561 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 17:36:01.998948 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998564 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 17:36:01.998948 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 17:36:01.998948 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:01.998895 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 17:36:02.003987 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.003966 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 17:36:02.005191 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.005171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c95477bc9-l7fk8"] Jul 27 17:36:02.041369 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.041345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:02.097489 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.097451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-config\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.097622 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.097509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxzj\" (UniqueName: \"kubernetes.io/projected/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-kube-api-access-6xxzj\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.097622 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.097558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-oauth-serving-cert\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.097622 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.097587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-service-ca\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.097622 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.097619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-serving-cert\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.097818 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.097634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-oauth-config\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.097818 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.097666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-trusted-ca-bundle\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.098304 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.098285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:02.198585 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.198550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-serving-cert\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.198585 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.198586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-oauth-config\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.198754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.198623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-trusted-ca-bundle\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.198754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.198668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-config\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.198754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.198701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxzj\" (UniqueName: \"kubernetes.io/projected/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-kube-api-access-6xxzj\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.198754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.198748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-oauth-serving-cert\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.198949 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.198778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-service-ca\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.199501 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.199477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-service-ca\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.199633 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.199610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-oauth-serving-cert\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.199678 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.199622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-config\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.199711 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.199674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-trusted-ca-bundle\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.201079 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.201058 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-oauth-config\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.201200 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.201185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-console-serving-cert\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.207621 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.207587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxzj\" (UniqueName: \"kubernetes.io/projected/ab2f07c2-8828-4767-a63e-2aa4928ffe2f-kube-api-access-6xxzj\") pod \"console-5c95477bc9-l7fk8\" (UID: \"ab2f07c2-8828-4767-a63e-2aa4928ffe2f\") " pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.303528 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.303432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:02.435800 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:02.435767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c95477bc9-l7fk8"] Jul 27 17:36:02.438705 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:36:02.438678 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2f07c2_8828_4767_a63e_2aa4928ffe2f.slice/crio-19848e6c5a8ba6061ae308432e7bbfed40ca93947cfde8ec1e3204a0ca9908b5 WatchSource:0}: Error finding container 19848e6c5a8ba6061ae308432e7bbfed40ca93947cfde8ec1e3204a0ca9908b5: Status 404 returned error can't find the container with id 19848e6c5a8ba6061ae308432e7bbfed40ca93947cfde8ec1e3204a0ca9908b5 Jul 27 17:36:03.323870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:03.323829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c95477bc9-l7fk8" event={"ID":"ab2f07c2-8828-4767-a63e-2aa4928ffe2f","Type":"ContainerStarted","Data":"1a60b3697a5d3e7bb3f6d90b042e7d1de590d0203c594f55bee4ecfc214d817a"} Jul 27 17:36:03.324252 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:03.323880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c95477bc9-l7fk8" event={"ID":"ab2f07c2-8828-4767-a63e-2aa4928ffe2f","Type":"ContainerStarted","Data":"19848e6c5a8ba6061ae308432e7bbfed40ca93947cfde8ec1e3204a0ca9908b5"} Jul 27 17:36:03.341309 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:03.341246 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c95477bc9-l7fk8" podStartSLOduration=2.341228739 podStartE2EDuration="2.341228739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:36:03.340767528 +0000 UTC m=+479.339553351" watchObservedRunningTime="2026-07-27 17:36:03.341228739 +0000 UTC m=+479.340014561" Jul 27 17:36:12.303857 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:12.303808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:12.304429 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:12.303901 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:12.308586 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:12.308563 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:12.361396 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:12.361365 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c95477bc9-l7fk8" Jul 27 17:36:13.020025 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:13.019984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:13.120251 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:13.120218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:14.257349 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:14.257319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:36:19.261619 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.261586 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bsbsw"] Jul 27 17:36:19.262024 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.261792 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" podUID="5d2bf4df-74a8-4311-8077-ac8e652878bd" containerName="manager" containerID="cri-o://ca341b246388efe3672f5ecafbb8fd847a477d5ca88a1666648ce1783beb5be4" gracePeriod=10 Jul 27 17:36:19.284865 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.284827 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-9fj98"] Jul 27 17:36:19.286992 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.286977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.295414 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.295391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-9fj98"] Jul 27 17:36:19.332731 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.332707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd37b378-d9d7-403e-a97b-a6702759db69-cert\") pod \"kserve-controller-manager-ff96d5959-9fj98\" (UID: \"fd37b378-d9d7-403e-a97b-a6702759db69\") " pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.332869 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.332740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grcd6\" (UniqueName: \"kubernetes.io/projected/fd37b378-d9d7-403e-a97b-a6702759db69-kube-api-access-grcd6\") pod \"kserve-controller-manager-ff96d5959-9fj98\" (UID: \"fd37b378-d9d7-403e-a97b-a6702759db69\") " pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.384668 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.384647 2564 generic.go:356] "Generic (PLEG): container finished" podID="5d2bf4df-74a8-4311-8077-ac8e652878bd" containerID="ca341b246388efe3672f5ecafbb8fd847a477d5ca88a1666648ce1783beb5be4" exitCode=0 Jul 27 17:36:19.384760 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.384706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" event={"ID":"5d2bf4df-74a8-4311-8077-ac8e652878bd","Type":"ContainerDied","Data":"ca341b246388efe3672f5ecafbb8fd847a477d5ca88a1666648ce1783beb5be4"} Jul 27 17:36:19.433846 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.433797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd37b378-d9d7-403e-a97b-a6702759db69-cert\") pod \"kserve-controller-manager-ff96d5959-9fj98\" (UID: \"fd37b378-d9d7-403e-a97b-a6702759db69\") " pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.433990 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.433860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grcd6\" (UniqueName: \"kubernetes.io/projected/fd37b378-d9d7-403e-a97b-a6702759db69-kube-api-access-grcd6\") pod \"kserve-controller-manager-ff96d5959-9fj98\" (UID: \"fd37b378-d9d7-403e-a97b-a6702759db69\") " pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.436159 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.436120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd37b378-d9d7-403e-a97b-a6702759db69-cert\") pod \"kserve-controller-manager-ff96d5959-9fj98\" (UID: \"fd37b378-d9d7-403e-a97b-a6702759db69\") " pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.443804 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.443751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grcd6\" (UniqueName: \"kubernetes.io/projected/fd37b378-d9d7-403e-a97b-a6702759db69-kube-api-access-grcd6\") pod \"kserve-controller-manager-ff96d5959-9fj98\" (UID: \"fd37b378-d9d7-403e-a97b-a6702759db69\") " pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.499178 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.499154 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:36:19.534785 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.534724 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2bf4df-74a8-4311-8077-ac8e652878bd-cert\") pod \"5d2bf4df-74a8-4311-8077-ac8e652878bd\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " Jul 27 17:36:19.534890 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.534823 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bbd8\" (UniqueName: \"kubernetes.io/projected/5d2bf4df-74a8-4311-8077-ac8e652878bd-kube-api-access-8bbd8\") pod \"5d2bf4df-74a8-4311-8077-ac8e652878bd\" (UID: \"5d2bf4df-74a8-4311-8077-ac8e652878bd\") " Jul 27 17:36:19.536981 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.536958 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2bf4df-74a8-4311-8077-ac8e652878bd-kube-api-access-8bbd8" (OuterVolumeSpecName: "kube-api-access-8bbd8") pod "5d2bf4df-74a8-4311-8077-ac8e652878bd" (UID: "5d2bf4df-74a8-4311-8077-ac8e652878bd"). InnerVolumeSpecName "kube-api-access-8bbd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:36:19.537135 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.537116 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2bf4df-74a8-4311-8077-ac8e652878bd-cert" (OuterVolumeSpecName: "cert") pod "5d2bf4df-74a8-4311-8077-ac8e652878bd" (UID: "5d2bf4df-74a8-4311-8077-ac8e652878bd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:36:19.635691 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.635665 2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2bf4df-74a8-4311-8077-ac8e652878bd-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:36:19.635691 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.635687 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbd8\" (UniqueName: \"kubernetes.io/projected/5d2bf4df-74a8-4311-8077-ac8e652878bd-kube-api-access-8bbd8\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:36:19.656668 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.656632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:19.778522 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:19.778431 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-9fj98"] Jul 27 17:36:19.780553 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:36:19.780531 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd37b378_d9d7_403e_a97b_a6702759db69.slice/crio-5b4e332513d4ac67e3d99b03dbe47f1e31ddafc6ad109b14316748c042b74161 WatchSource:0}: Error finding container 5b4e332513d4ac67e3d99b03dbe47f1e31ddafc6ad109b14316748c042b74161: Status 404 returned error can't find the container with id 5b4e332513d4ac67e3d99b03dbe47f1e31ddafc6ad109b14316748c042b74161 Jul 27 17:36:20.389090 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:20.389063 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" Jul 27 17:36:20.389090 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:20.389082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-bsbsw" event={"ID":"5d2bf4df-74a8-4311-8077-ac8e652878bd","Type":"ContainerDied","Data":"5170983beb7dbfb555f7ad3de9ad61875ee4db7056cee4730e0459a3ab6c1247"} Jul 27 17:36:20.389545 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:20.389162 2564 scope.go:117] "RemoveContainer" containerID="ca341b246388efe3672f5ecafbb8fd847a477d5ca88a1666648ce1783beb5be4" Jul 27 17:36:20.390463 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:20.390318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" event={"ID":"fd37b378-d9d7-403e-a97b-a6702759db69","Type":"ContainerStarted","Data":"5b4e332513d4ac67e3d99b03dbe47f1e31ddafc6ad109b14316748c042b74161"} Jul 27 17:36:20.410978 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:20.410948 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bsbsw"] Jul 27 17:36:20.414686 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:20.414667 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bsbsw"] Jul 27 17:36:20.504331 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:20.504305 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2bf4df-74a8-4311-8077-ac8e652878bd" path="/var/lib/kubelet/pods/5d2bf4df-74a8-4311-8077-ac8e652878bd/volumes" Jul 27 17:36:21.401255 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:21.401221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" event={"ID":"fd37b378-d9d7-403e-a97b-a6702759db69","Type":"ContainerStarted","Data":"d64432a5f115c580c5f6a10a2c889bd38397d734bec71e33c5198aa0d6d7ec60"} Jul 27 17:36:21.401627 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:21.401364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:21.422066 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:21.422024 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" podStartSLOduration=1.737583162 podStartE2EDuration="2.422012576s" totalImagesPullingTime="684.429414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:36:21.420184861 +0000 UTC m=+497.418970701" watchObservedRunningTime="2026-07-27 17:36:21.422012576 +0000 UTC m=+497.420798396" Jul 27 17:36:52.410044 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:52.410012 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 17:36:53.382271 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.382216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:53.434412 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.434379 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tjfjs"] Jul 27 17:36:53.434851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.434799 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2bf4df-74a8-4311-8077-ac8e652878bd" containerName="manager" Jul 27 17:36:53.434851 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.434812 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2bf4df-74a8-4311-8077-ac8e652878bd" containerName="manager" Jul 27 17:36:53.434958 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.434917 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2bf4df-74a8-4311-8077-ac8e652878bd" containerName="manager" Jul 27 17:36:53.437128 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.437111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.439837 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.439817 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 27 17:36:53.440097 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.440079 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-h5lxj\"" Jul 27 17:36:53.447700 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.447670 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tjfjs"] Jul 27 17:36:53.460267 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.460245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:53.564490 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.564464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2def4a9c-07d8-4703-910d-3e48801c3b2b-cert\") pod \"odh-model-controller-75d9cdbddf-tjfjs\" (UID: \"2def4a9c-07d8-4703-910d-3e48801c3b2b\") " pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.564653 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.564513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jrd\" (UniqueName: \"kubernetes.io/projected/2def4a9c-07d8-4703-910d-3e48801c3b2b-kube-api-access-h4jrd\") pod \"odh-model-controller-75d9cdbddf-tjfjs\" (UID: \"2def4a9c-07d8-4703-910d-3e48801c3b2b\") " pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.644048 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.643963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:53.665737 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.665699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jrd\" (UniqueName: \"kubernetes.io/projected/2def4a9c-07d8-4703-910d-3e48801c3b2b-kube-api-access-h4jrd\") pod \"odh-model-controller-75d9cdbddf-tjfjs\" (UID: \"2def4a9c-07d8-4703-910d-3e48801c3b2b\") " pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.665878 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.665769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2def4a9c-07d8-4703-910d-3e48801c3b2b-cert\") pod \"odh-model-controller-75d9cdbddf-tjfjs\" (UID: \"2def4a9c-07d8-4703-910d-3e48801c3b2b\") " pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.667973 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.667952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2def4a9c-07d8-4703-910d-3e48801c3b2b-cert\") pod \"odh-model-controller-75d9cdbddf-tjfjs\" (UID: \"2def4a9c-07d8-4703-910d-3e48801c3b2b\") " pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.674660 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.674640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jrd\" (UniqueName: \"kubernetes.io/projected/2def4a9c-07d8-4703-910d-3e48801c3b2b-kube-api-access-h4jrd\") pod \"odh-model-controller-75d9cdbddf-tjfjs\" (UID: \"2def4a9c-07d8-4703-910d-3e48801c3b2b\") " pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.744892 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.744862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:53.749822 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.749800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:53.870570 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:53.870538 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-tjfjs"] Jul 27 17:36:53.873589 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:36:53.873561 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2def4a9c_07d8_4703_910d_3e48801c3b2b.slice/crio-528a751f273231659385ed9d24b2081efd717ba59ee33bf55697701b2de29624 WatchSource:0}: Error finding container 528a751f273231659385ed9d24b2081efd717ba59ee33bf55697701b2de29624: Status 404 returned error can't find the container with id 528a751f273231659385ed9d24b2081efd717ba59ee33bf55697701b2de29624 Jul 27 17:36:54.047702 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:54.047668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:54.146758 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:54.146727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:36:54.527247 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:54.527184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" event={"ID":"2def4a9c-07d8-4703-910d-3e48801c3b2b","Type":"ContainerStarted","Data":"528a751f273231659385ed9d24b2081efd717ba59ee33bf55697701b2de29624"} Jul 27 17:36:57.542292 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:57.542255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" event={"ID":"2def4a9c-07d8-4703-910d-3e48801c3b2b","Type":"ContainerStarted","Data":"9696fa11034dd4b44388d229c0bdc402341eefaaa21b3456b441d785e5dd559c"} Jul 27 17:36:57.542665 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:57.542373 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:36:57.563457 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:36:57.563411 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" podStartSLOduration=1.507217051 podStartE2EDuration="4.563398043s" totalImagesPullingTime="3.056180992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:36:57.561347422 +0000 UTC m=+533.560133264" watchObservedRunningTime="2026-07-27 17:36:57.563398043 +0000 UTC m=+533.562183865" Jul 27 17:37:02.657870 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:02.657833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:37:08.548434 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:08.548407 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-tjfjs" Jul 27 17:37:09.378593 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.378565 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-svwhx"] Jul 27 17:37:09.386300 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.386269 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-svwhx" Jul 27 17:37:09.389725 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.389701 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-svwhx"] Jul 27 17:37:09.497462 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.497435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx89x\" (UniqueName: \"kubernetes.io/projected/1712a34c-40a4-40cc-841a-e2453c7028b2-kube-api-access-rx89x\") pod \"s3-init-svwhx\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " pod="kserve/s3-init-svwhx" Jul 27 17:37:09.497593 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.497476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a34c-40a4-40cc-841a-e2453c7028b2-model-cache\") pod \"s3-init-svwhx\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " pod="kserve/s3-init-svwhx" Jul 27 17:37:09.598447 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.598424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx89x\" (UniqueName: \"kubernetes.io/projected/1712a34c-40a4-40cc-841a-e2453c7028b2-kube-api-access-rx89x\") pod \"s3-init-svwhx\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " pod="kserve/s3-init-svwhx" Jul 27 17:37:09.598764 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.598467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a34c-40a4-40cc-841a-e2453c7028b2-model-cache\") pod \"s3-init-svwhx\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " pod="kserve/s3-init-svwhx" Jul 27 17:37:09.598855 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.598838 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a34c-40a4-40cc-841a-e2453c7028b2-model-cache\") pod \"s3-init-svwhx\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " pod="kserve/s3-init-svwhx" Jul 27 17:37:09.607708 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.607688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx89x\" (UniqueName: \"kubernetes.io/projected/1712a34c-40a4-40cc-841a-e2453c7028b2-kube-api-access-rx89x\") pod \"s3-init-svwhx\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " pod="kserve/s3-init-svwhx" Jul 27 17:37:09.706785 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.706736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-svwhx" Jul 27 17:37:09.831467 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:09.831441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-svwhx"] Jul 27 17:37:09.834063 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:37:09.834031 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1712a34c_40a4_40cc_841a_e2453c7028b2.slice/crio-83aeb3b6063a117498815b63aeaac830529f28f877d28e9419fbbd7c7a87bbdb WatchSource:0}: Error finding container 83aeb3b6063a117498815b63aeaac830529f28f877d28e9419fbbd7c7a87bbdb: Status 404 returned error can't find the container with id 83aeb3b6063a117498815b63aeaac830529f28f877d28e9419fbbd7c7a87bbdb Jul 27 17:37:10.592640 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:10.592601 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-svwhx" event={"ID":"1712a34c-40a4-40cc-841a-e2453c7028b2","Type":"ContainerStarted","Data":"83aeb3b6063a117498815b63aeaac830529f28f877d28e9419fbbd7c7a87bbdb"} Jul 27 17:37:13.606809 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:13.606761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-svwhx" event={"ID":"1712a34c-40a4-40cc-841a-e2453c7028b2","Type":"ContainerStarted","Data":"4a30e4f90160e982127675e93ff6e1256010983e99f409750cbc19d5c41b3abd"} Jul 27 17:37:16.619752 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:16.619716 2564 generic.go:356] "Generic (PLEG): container finished" podID="1712a34c-40a4-40cc-841a-e2453c7028b2" containerID="4a30e4f90160e982127675e93ff6e1256010983e99f409750cbc19d5c41b3abd" exitCode=0 Jul 27 17:37:16.620126 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:16.619764 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-svwhx" event={"ID":"1712a34c-40a4-40cc-841a-e2453c7028b2","Type":"ContainerDied","Data":"4a30e4f90160e982127675e93ff6e1256010983e99f409750cbc19d5c41b3abd"} Jul 27 17:37:21.645466 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:21.645381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-svwhx" event={"ID":"1712a34c-40a4-40cc-841a-e2453c7028b2","Type":"ContainerStarted","Data":"4d2d46a7a0a918babd19d6e23f458f7f7b29c206b3a641160376785c379ebdcd"} Jul 27 17:37:21.662897 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:37:21.662837 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-svwhx" podStartSLOduration=1.52210019 podStartE2EDuration="12.662817875s" totalImagesPullingTime="8.251275605s" totalInitContainerRuntime="2.88944208s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:21.661569746 +0000 UTC m=+557.660355576" watchObservedRunningTime="2026-07-27 17:37:21.662817875 +0000 UTC m=+557.661603697" Jul 27 17:38:12.645873 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:12.645839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:26.887251 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:26.887214 2564 generic.go:356] "Generic (PLEG): container finished" podID="1712a34c-40a4-40cc-841a-e2453c7028b2" containerID="4d2d46a7a0a918babd19d6e23f458f7f7b29c206b3a641160376785c379ebdcd" exitCode=0 Jul 27 17:38:26.887754 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:26.887262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-svwhx" event={"ID":"1712a34c-40a4-40cc-841a-e2453c7028b2","Type":"ContainerDied","Data":"4d2d46a7a0a918babd19d6e23f458f7f7b29c206b3a641160376785c379ebdcd"} Jul 27 17:38:28.010093 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.010067 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-svwhx" Jul 27 17:38:28.123674 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.123628 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx89x\" (UniqueName: \"kubernetes.io/projected/1712a34c-40a4-40cc-841a-e2453c7028b2-kube-api-access-rx89x\") pod \"1712a34c-40a4-40cc-841a-e2453c7028b2\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " Jul 27 17:38:28.123882 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.123720 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a34c-40a4-40cc-841a-e2453c7028b2-model-cache\") pod \"1712a34c-40a4-40cc-841a-e2453c7028b2\" (UID: \"1712a34c-40a4-40cc-841a-e2453c7028b2\") " Jul 27 17:38:28.125696 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.125661 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1712a34c-40a4-40cc-841a-e2453c7028b2-kube-api-access-rx89x" (OuterVolumeSpecName: "kube-api-access-rx89x") pod "1712a34c-40a4-40cc-841a-e2453c7028b2" (UID: "1712a34c-40a4-40cc-841a-e2453c7028b2"). InnerVolumeSpecName "kube-api-access-rx89x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:38:28.142013 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.141953 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712a34c-40a4-40cc-841a-e2453c7028b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1712a34c-40a4-40cc-841a-e2453c7028b2" (UID: "1712a34c-40a4-40cc-841a-e2453c7028b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:38:28.225193 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.225158 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a34c-40a4-40cc-841a-e2453c7028b2-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:38:28.225193 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.225188 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx89x\" (UniqueName: \"kubernetes.io/projected/1712a34c-40a4-40cc-841a-e2453c7028b2-kube-api-access-rx89x\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:38:28.896219 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.896125 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-svwhx" Jul 27 17:38:28.896219 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.896160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-svwhx" event={"ID":"1712a34c-40a4-40cc-841a-e2453c7028b2","Type":"ContainerDied","Data":"83aeb3b6063a117498815b63aeaac830529f28f877d28e9419fbbd7c7a87bbdb"} Jul 27 17:38:28.896219 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:28.896194 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83aeb3b6063a117498815b63aeaac830529f28f877d28e9419fbbd7c7a87bbdb" Jul 27 17:38:29.249409 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:29.249378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:29.349787 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:29.349757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.437674 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.437639 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.743957 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.743923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.778687 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.778658 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x"] Jul 27 17:38:40.778985 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.778971 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1712a34c-40a4-40cc-841a-e2453c7028b2" containerName="download-hf-model" Jul 27 17:38:40.779041 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.778990 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1712a34c-40a4-40cc-841a-e2453c7028b2" containerName="download-hf-model" Jul 27 17:38:40.779041 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.779010 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1712a34c-40a4-40cc-841a-e2453c7028b2" containerName="s3-init" Jul 27 17:38:40.779041 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.779015 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1712a34c-40a4-40cc-841a-e2453c7028b2" containerName="s3-init" Jul 27 17:38:40.779134 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.779065 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1712a34c-40a4-40cc-841a-e2453c7028b2" containerName="s3-init" Jul 27 17:38:40.782326 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.782308 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.785163 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.785124 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" Jul 27 17:38:40.785282 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.785232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 27 17:38:40.785482 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.785465 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 27 17:38:40.785571 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.785529 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 27 17:38:40.785571 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.785540 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" Jul 27 17:38:40.785836 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.785819 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 17:38:40.786902 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.786884 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x6c7x\"" Jul 27 17:38:40.792715 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.792693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x"] Jul 27 17:38:40.827611 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.827585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72cb5b5d-6976-4493-a299-27c402e96ab4-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.827734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.827624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.827734 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.827664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.827820 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.827738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsfm\" (UniqueName: \"kubernetes.io/projected/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-api-access-dlsfm\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.827861 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.827813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.866759 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.866729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.928765 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.928735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.928937 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.928773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsfm\" (UniqueName: \"kubernetes.io/projected/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-api-access-dlsfm\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.928937 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.928889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.928937 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:38:40.928919 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Jul 27 17:38:40.929094 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:38:40.929007 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls podName:72cb5b5d-6976-4493-a299-27c402e96ab4 nodeName:}" failed. No retries permitted until 2026-07-27 17:38:41.428986449 +0000 UTC m=+637.427772263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls") pod "isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" (UID: "72cb5b5d-6976-4493-a299-27c402e96ab4") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Jul 27 17:38:40.929094 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.928930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72cb5b5d-6976-4493-a299-27c402e96ab4-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.929230 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.929117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.929325 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.929300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72cb5b5d-6976-4493-a299-27c402e96ab4-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.929932 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.929906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.930116 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.930008 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.936113 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.936075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:40.938325 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.938291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsfm\" (UniqueName: \"kubernetes.io/projected/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-api-access-dlsfm\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:40.986526 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.986491 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6"] Jul 27 17:38:40.989905 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.989887 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:40.992665 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.992644 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Jul 27 17:38:40.992780 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:40.992642 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Jul 27 17:38:41.000839 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.000815 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6"] Jul 27 17:38:41.002981 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.002957 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:41.029916 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.029888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6x9\" (UniqueName: \"kubernetes.io/projected/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-api-access-cf6x9\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.030064 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.029937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.030064 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.030019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.030064 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.030060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff1b82a-f995-47c1-9523-83ee56140b7b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.030254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.030090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff1b82a-f995-47c1-9523-83ee56140b7b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.070894 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.070831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:41.086766 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.086733 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68"] Jul 27 17:38:41.089984 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.089964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.092794 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.092756 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Jul 27 17:38:41.092794 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.092783 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Jul 27 17:38:41.102187 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.102165 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68"] Jul 27 17:38:41.131392 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6x9\" (UniqueName: \"kubernetes.io/projected/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-api-access-cf6x9\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.131392 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.131558 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.131558 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094d3dc7-e47a-49c2-892e-f1631d9eba0b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.131558 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.131704 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.131704 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff1b82a-f995-47c1-9523-83ee56140b7b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.131704 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf9f\" (UniqueName: \"kubernetes.io/projected/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-api-access-vpf9f\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.131704 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.131704 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.131693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff1b82a-f995-47c1-9523-83ee56140b7b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.132503 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.132475 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.132635 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.132484 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.132704 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.132657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff1b82a-f995-47c1-9523-83ee56140b7b-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.132832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.132813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:41.134197 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.134167 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff1b82a-f995-47c1-9523-83ee56140b7b-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.140683 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.140659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6x9\" (UniqueName: \"kubernetes.io/projected/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-api-access-cf6x9\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6mzj6\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.225812 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.225777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:41.232927 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.232903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094d3dc7-e47a-49c2-892e-f1631d9eba0b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.233040 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.232951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.233094 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.233056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf9f\" (UniqueName: \"kubernetes.io/projected/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-api-access-vpf9f\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.233164 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.233097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.233229 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.233198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.233564 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.233539 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.233653 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.233639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.233715 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.233696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.235359 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.235337 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094d3dc7-e47a-49c2-892e-f1631d9eba0b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.244285 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.244259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf9f\" (UniqueName: \"kubernetes.io/projected/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-api-access-vpf9f\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-rmd68\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.300333 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.300268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:38:41.301170 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.301136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:38:41.401793 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.401760 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:38:41.435223 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.435190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:41.439136 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.439080 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:41.441640 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.441615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6"] Jul 27 17:38:41.442674 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:38:41.442647 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff1b82a_f995_47c1_9523_83ee56140b7b.slice/crio-0d2c4f686b08f08a982c0674960f26b8df65ff699fca3599cec0f7be8566b5e8 WatchSource:0}: Error finding container 0d2c4f686b08f08a982c0674960f26b8df65ff699fca3599cec0f7be8566b5e8: Status 404 returned error can't find the container with id 0d2c4f686b08f08a982c0674960f26b8df65ff699fca3599cec0f7be8566b5e8 Jul 27 17:38:41.540768 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.540736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68"] Jul 27 17:38:41.544299 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:38:41.544272 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094d3dc7_e47a_49c2_892e_f1631d9eba0b.slice/crio-2803e7e4a9bbb416b438c0b3369f3d432a744e53185cea064aa72f989c04b946 WatchSource:0}: Error finding container 2803e7e4a9bbb416b438c0b3369f3d432a744e53185cea064aa72f989c04b946: Status 404 returned error can't find the container with id 2803e7e4a9bbb416b438c0b3369f3d432a744e53185cea064aa72f989c04b946 Jul 27 17:38:41.695296 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.695257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:38:41.838912 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.838839 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x"] Jul 27 17:38:41.841216 ip-10-0-131-71 kubenswrapper[2564]: W0727 17:38:41.841186 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cb5b5d_6976_4493_a299_27c402e96ab4.slice/crio-b949e815ee2fee86ed7dfae605d6fa52edac2095d0f6bdb3d6a77417aa7e1fd4 WatchSource:0}: Error finding container b949e815ee2fee86ed7dfae605d6fa52edac2095d0f6bdb3d6a77417aa7e1fd4: Status 404 returned error can't find the container with id b949e815ee2fee86ed7dfae605d6fa52edac2095d0f6bdb3d6a77417aa7e1fd4 Jul 27 17:38:41.965274 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.964815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerStarted","Data":"be919116b1d6e5790a1364b9eb88175182ee9bfa94e4a06cb57547bfbfda6fc5"} Jul 27 17:38:41.965274 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.964926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerStarted","Data":"b949e815ee2fee86ed7dfae605d6fa52edac2095d0f6bdb3d6a77417aa7e1fd4"} Jul 27 17:38:41.966826 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.966723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerStarted","Data":"17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4"} Jul 27 17:38:41.966826 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.966786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerStarted","Data":"0d2c4f686b08f08a982c0674960f26b8df65ff699fca3599cec0f7be8566b5e8"} Jul 27 17:38:41.968718 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.968687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerStarted","Data":"0d461eb986c92f940230f5f4fcfd4e70bd5793c8bd13f522551288748b163af1"} Jul 27 17:38:41.968854 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:41.968749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerStarted","Data":"2803e7e4a9bbb416b438c0b3369f3d432a744e53185cea064aa72f989c04b946"} Jul 27 17:38:45.985551 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:45.985512 2564 generic.go:356] "Generic (PLEG): container finished" podID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerID="0d461eb986c92f940230f5f4fcfd4e70bd5793c8bd13f522551288748b163af1" exitCode=0 Jul 27 17:38:45.990319 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:45.985574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerDied","Data":"0d461eb986c92f940230f5f4fcfd4e70bd5793c8bd13f522551288748b163af1"} Jul 27 17:38:45.990319 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:45.987388 2564 generic.go:356] "Generic (PLEG): container finished" podID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerID="be919116b1d6e5790a1364b9eb88175182ee9bfa94e4a06cb57547bfbfda6fc5" exitCode=0 Jul 27 17:38:45.990319 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:45.987463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerDied","Data":"be919116b1d6e5790a1364b9eb88175182ee9bfa94e4a06cb57547bfbfda6fc5"} Jul 27 17:38:45.990319 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:45.989037 2564 generic.go:356] "Generic (PLEG): container finished" podID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerID="17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4" exitCode=0 Jul 27 17:38:45.990319 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:45.989079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerDied","Data":"17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4"} Jul 27 17:38:58.263161 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:38:58.263042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:39:00.208828 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:00.208796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:39:15.185733 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:15.185698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerStarted","Data":"1e59920fabfac22f1edb1a94b2ef5d6db3043f0f99027018da3ac0a790eaddec"} Jul 27 17:39:15.187562 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:15.187537 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerStarted","Data":"cfea891e9a06b4685d1fe368f936d9b55c2aeb0488387b4e1919432f04d07c10"} Jul 27 17:39:16.197466 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:16.197330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerStarted","Data":"17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c"} Jul 27 17:39:16.653310 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:16.653208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:39:18.208246 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.208206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerStarted","Data":"73574124dece82b31e9a43035e7a686e6e6eee3e1b2b7262dcae2db697f578e1"} Jul 27 17:39:18.208667 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.208334 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:39:18.210187 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.210158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerStarted","Data":"00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600"} Jul 27 17:39:18.210331 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.210217 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:39:18.210331 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.210265 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:39:18.211623 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.211601 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:39:18.212373 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.212355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerStarted","Data":"664e02fe69f7fea5f385065e602db3f74525a4bb99d026bbf715ecc60d629084"} Jul 27 17:39:18.212505 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.212489 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:39:18.212548 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.212540 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:39:18.213540 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.213512 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:39:18.232168 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.232111 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podStartSLOduration=2.855289165 podStartE2EDuration="38.232097962s" totalImagesPullingTime="31.550207487s" totalInitContainerRuntime="3.82660131s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:18.230815796 +0000 UTC m=+674.229601631" watchObservedRunningTime="2026-07-27 17:39:18.232097962 +0000 UTC m=+674.230883782" Jul 27 17:39:18.249935 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.249883 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podStartSLOduration=2.829364344 podStartE2EDuration="38.249867446s" totalImagesPullingTime="31.565433858s" totalInitContainerRuntime="3.855069244s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:18.249004144 +0000 UTC m=+674.247789993" watchObservedRunningTime="2026-07-27 17:39:18.249867446 +0000 UTC m=+674.248653267" Jul 27 17:39:18.268955 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:18.268910 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podStartSLOduration=2.023844669 podStartE2EDuration="37.268893643s" totalImagesPullingTime="31.538704843s" totalInitContainerRuntime="3.706344131s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:18.267099236 +0000 UTC m=+674.265885094" watchObservedRunningTime="2026-07-27 17:39:18.268893643 +0000 UTC m=+674.267679464" Jul 27 17:39:19.216131 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:19.216082 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:39:19.216536 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:19.216194 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:39:19.216603 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:19.216585 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:39:19.217532 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:19.217507 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:39:20.219911 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:20.219869 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:39:21.763045 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:21.762987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:39:21.823695 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:21.823662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:39:24.220864 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:24.220836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:39:24.221272 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:24.221241 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:39:24.221370 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:24.221342 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:39:24.221702 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:24.221678 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:39:25.224488 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:25.224462 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:39:25.224894 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:25.224867 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:39:34.221950 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:34.221905 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:39:34.222336 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:34.221905 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:39:35.224931 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:35.224887 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:39:44.221982 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:44.221938 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:39:44.222393 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:44.221938 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:39:45.225034 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:45.224997 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:39:54.221639 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:54.221591 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:39:54.222116 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:54.221608 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:39:55.225337 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:39:55.225294 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:40:00.481457 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:00.481371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:00.806922 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:00.806795 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:04.221505 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:04.221464 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:40:04.221886 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:04.221688 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:40:05.225407 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:05.225364 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:40:10.611453 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:10.611414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:10.726368 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:10.726299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:10.802933 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:10.802892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:10.873242 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:10.873133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:11.142247 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:11.142165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:11.381628 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:11.381592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:11.483506 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:11.483477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:40:14.222161 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:14.222101 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:40:14.222161 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:14.222101 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:40:15.225583 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:15.225542 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:40:24.222128 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:24.222089 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:40:24.222524 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:24.222274 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:40:25.225264 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:25.225228 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:40:34.221980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:34.221950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:40:35.225318 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:40:35.225281 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:41:00.580088 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:00.580052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:00.772448 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:00.772414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:10.699819 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.699783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:10.727194 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.725063 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68"] Jul 27 17:41:10.727194 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.725543 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" containerID="cri-o://1e59920fabfac22f1edb1a94b2ef5d6db3043f0f99027018da3ac0a790eaddec" gracePeriod=30 Jul 27 17:41:10.727194 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.726006 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kube-rbac-proxy" containerID="cri-o://664e02fe69f7fea5f385065e602db3f74525a4bb99d026bbf715ecc60d629084" gracePeriod=30 Jul 27 17:41:10.785038 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.785005 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x"] Jul 27 17:41:10.785572 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.785541 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" containerID="cri-o://cfea891e9a06b4685d1fe368f936d9b55c2aeb0488387b4e1919432f04d07c10" gracePeriod=30 Jul 27 17:41:10.785879 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.785742 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kube-rbac-proxy" containerID="cri-o://73574124dece82b31e9a43035e7a686e6e6eee3e1b2b7262dcae2db697f578e1" gracePeriod=30 Jul 27 17:41:10.791492 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.791402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:10.866967 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.866681 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6"] Jul 27 17:41:10.867299 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.867245 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kube-rbac-proxy" containerID="cri-o://00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600" gracePeriod=30 Jul 27 17:41:10.868011 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.867013 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" containerID="cri-o://17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c" gracePeriod=30 Jul 27 17:41:10.871379 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.871338 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:10.972214 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:10.972097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:11.072740 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.072710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:11.536680 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.536640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:41:11.686018 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.685987 2564 generic.go:356] "Generic (PLEG): container finished" podID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerID="73574124dece82b31e9a43035e7a686e6e6eee3e1b2b7262dcae2db697f578e1" exitCode=2 Jul 27 17:41:11.686200 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.686055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerDied","Data":"73574124dece82b31e9a43035e7a686e6e6eee3e1b2b7262dcae2db697f578e1"} Jul 27 17:41:11.688001 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.687976 2564 generic.go:356] "Generic (PLEG): container finished" podID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerID="00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600" exitCode=2 Jul 27 17:41:11.688116 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.688043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerDied","Data":"00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600"} Jul 27 17:41:11.690174 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.690153 2564 generic.go:356] "Generic (PLEG): container finished" podID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerID="664e02fe69f7fea5f385065e602db3f74525a4bb99d026bbf715ecc60d629084" exitCode=2 Jul 27 17:41:11.690273 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:11.690176 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerDied","Data":"664e02fe69f7fea5f385065e602db3f74525a4bb99d026bbf715ecc60d629084"} Jul 27 17:41:14.216888 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.216848 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused" Jul 27 17:41:14.217286 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.216849 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused" Jul 27 17:41:14.222268 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.222236 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Jul 27 17:41:14.222382 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.222235 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Jul 27 17:41:14.623282 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.623259 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:41:14.704577 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.704545 2564 generic.go:356] "Generic (PLEG): container finished" podID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerID="17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c" exitCode=0 Jul 27 17:41:14.704740 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.704611 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" Jul 27 17:41:14.704740 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.704628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerDied","Data":"17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c"} Jul 27 17:41:14.704740 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.704700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6" event={"ID":"5ff1b82a-f995-47c1-9523-83ee56140b7b","Type":"ContainerDied","Data":"0d2c4f686b08f08a982c0674960f26b8df65ff699fca3599cec0f7be8566b5e8"} Jul 27 17:41:14.704740 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.704735 2564 scope.go:117] "RemoveContainer" containerID="00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600" Jul 27 17:41:14.712639 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.712621 2564 scope.go:117] "RemoveContainer" containerID="17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c" Jul 27 17:41:14.719872 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.719857 2564 scope.go:117] "RemoveContainer" containerID="17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4" Jul 27 17:41:14.726736 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.726722 2564 scope.go:117] "RemoveContainer" containerID="00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600" Jul 27 17:41:14.726972 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:41:14.726953 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600\": container with ID starting with 00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600 not found: ID does not exist" containerID="00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600" Jul 27 17:41:14.727031 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.726981 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600"} err="failed to get container status \"00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600\": rpc error: code = NotFound desc = could not find container \"00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600\": container with ID starting with 00318b41fbff67afc09481f6276cbfa2eaacced1f2e031287d1bc4e8e8792600 not found: ID does not exist" Jul 27 17:41:14.727031 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.726999 2564 scope.go:117] "RemoveContainer" containerID="17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c" Jul 27 17:41:14.727240 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:41:14.727225 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c\": container with ID starting with 17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c not found: ID does not exist" containerID="17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c" Jul 27 17:41:14.727311 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.727246 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c"} err="failed to get container status \"17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c\": rpc error: code = NotFound desc = could not find container \"17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c\": container with ID starting with 17d5605b147e6ac4cc836a544a7d0ece00f0b2445122ad3ce29e608d8df60c9c not found: ID does not exist" Jul 27 17:41:14.727311 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.727260 2564 scope.go:117] "RemoveContainer" containerID="17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4" Jul 27 17:41:14.727490 ip-10-0-131-71 kubenswrapper[2564]: E0727 17:41:14.727476 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4\": container with ID starting with 17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4 not found: ID does not exist" containerID="17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4" Jul 27 17:41:14.727533 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.727493 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4"} err="failed to get container status \"17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4\": rpc error: code = NotFound desc = could not find container \"17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4\": container with ID starting with 17e9f2a9655f762652988ec98472ea797ed857be4d4bb0866dd50b4cc071f0b4 not found: ID does not exist" Jul 27 17:41:14.784307 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-cabundle-cert\") pod \"5ff1b82a-f995-47c1-9523-83ee56140b7b\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " Jul 27 17:41:14.784307 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784271 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-rbac-proxy-sar-config\") pod \"5ff1b82a-f995-47c1-9523-83ee56140b7b\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " Jul 27 17:41:14.784307 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784305 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff1b82a-f995-47c1-9523-83ee56140b7b-kserve-provision-location\") pod \"5ff1b82a-f995-47c1-9523-83ee56140b7b\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " Jul 27 17:41:14.784557 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784353 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf6x9\" (UniqueName: \"kubernetes.io/projected/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-api-access-cf6x9\") pod \"5ff1b82a-f995-47c1-9523-83ee56140b7b\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " Jul 27 17:41:14.784557 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff1b82a-f995-47c1-9523-83ee56140b7b-proxy-tls\") pod \"5ff1b82a-f995-47c1-9523-83ee56140b7b\" (UID: \"5ff1b82a-f995-47c1-9523-83ee56140b7b\") " Jul 27 17:41:14.784683 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784652 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ff1b82a-f995-47c1-9523-83ee56140b7b" (UID: "5ff1b82a-f995-47c1-9523-83ee56140b7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:41:14.784731 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784662 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ff1b82a-f995-47c1-9523-83ee56140b7b" (UID: "5ff1b82a-f995-47c1-9523-83ee56140b7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:41:14.784731 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.784666 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff1b82a-f995-47c1-9523-83ee56140b7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ff1b82a-f995-47c1-9523-83ee56140b7b" (UID: "5ff1b82a-f995-47c1-9523-83ee56140b7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:41:14.786618 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.786589 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff1b82a-f995-47c1-9523-83ee56140b7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ff1b82a-f995-47c1-9523-83ee56140b7b" (UID: "5ff1b82a-f995-47c1-9523-83ee56140b7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:41:14.786713 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.786641 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-api-access-cf6x9" (OuterVolumeSpecName: "kube-api-access-cf6x9") pod "5ff1b82a-f995-47c1-9523-83ee56140b7b" (UID: "5ff1b82a-f995-47c1-9523-83ee56140b7b"). InnerVolumeSpecName "kube-api-access-cf6x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:41:14.885107 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.885073 2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-cabundle-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:14.885107 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.885100 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:14.885107 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.885109 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff1b82a-f995-47c1-9523-83ee56140b7b-kserve-provision-location\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:14.885349 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.885120 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6x9\" (UniqueName: \"kubernetes.io/projected/5ff1b82a-f995-47c1-9523-83ee56140b7b-kube-api-access-cf6x9\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:14.885349 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:14.885131 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff1b82a-f995-47c1-9523-83ee56140b7b-proxy-tls\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:15.026831 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.026803 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6"] Jul 27 17:41:15.030774 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.030751 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6mzj6"] Jul 27 17:41:15.220629 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.220527 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused" Jul 27 17:41:15.224954 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.224924 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Jul 27 17:41:15.712979 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.712947 2564 generic.go:356] "Generic (PLEG): container finished" podID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerID="cfea891e9a06b4685d1fe368f936d9b55c2aeb0488387b4e1919432f04d07c10" exitCode=0 Jul 27 17:41:15.713156 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.713024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerDied","Data":"cfea891e9a06b4685d1fe368f936d9b55c2aeb0488387b4e1919432f04d07c10"} Jul 27 17:41:15.716256 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.716234 2564 generic.go:356] "Generic (PLEG): container finished" podID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerID="1e59920fabfac22f1edb1a94b2ef5d6db3043f0f99027018da3ac0a790eaddec" exitCode=0 Jul 27 17:41:15.716368 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.716273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerDied","Data":"1e59920fabfac22f1edb1a94b2ef5d6db3043f0f99027018da3ac0a790eaddec"} Jul 27 17:41:15.800917 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.800887 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:41:15.803796 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.803774 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:41:15.994832 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994795 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-rbac-proxy-sar-config\") pod \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " Jul 27 17:41:15.995009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994851 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72cb5b5d-6976-4493-a299-27c402e96ab4-kserve-provision-location\") pod \"72cb5b5d-6976-4493-a299-27c402e96ab4\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " Jul 27 17:41:15.995009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994876 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094d3dc7-e47a-49c2-892e-f1631d9eba0b-proxy-tls\") pod \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " Jul 27 17:41:15.995009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994901 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-cabundle-cert\") pod \"72cb5b5d-6976-4493-a299-27c402e96ab4\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " Jul 27 17:41:15.995009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994918 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kserve-provision-location\") pod \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " Jul 27 17:41:15.995009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994954 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls\") pod \"72cb5b5d-6976-4493-a299-27c402e96ab4\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " Jul 27 17:41:15.995009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpf9f\" (UniqueName: \"kubernetes.io/projected/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-api-access-vpf9f\") pod \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " Jul 27 17:41:15.995009 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.994997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlsfm\" (UniqueName: \"kubernetes.io/projected/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-api-access-dlsfm\") pod \"72cb5b5d-6976-4493-a299-27c402e96ab4\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " Jul 27 17:41:15.995401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995027 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-rbac-proxy-sar-config\") pod \"72cb5b5d-6976-4493-a299-27c402e96ab4\" (UID: \"72cb5b5d-6976-4493-a299-27c402e96ab4\") " Jul 27 17:41:15.995401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-cabundle-cert\") pod \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\" (UID: \"094d3dc7-e47a-49c2-892e-f1631d9eba0b\") " Jul 27 17:41:15.995401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995261 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cb5b5d-6976-4493-a299-27c402e96ab4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72cb5b5d-6976-4493-a299-27c402e96ab4" (UID: "72cb5b5d-6976-4493-a299-27c402e96ab4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:41:15.995401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995290 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "094d3dc7-e47a-49c2-892e-f1631d9eba0b" (UID: "094d3dc7-e47a-49c2-892e-f1631d9eba0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:41:15.995401 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995289 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72cb5b5d-6976-4493-a299-27c402e96ab4" (UID: "72cb5b5d-6976-4493-a299-27c402e96ab4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:41:15.995633 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995508 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:15.995633 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995530 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72cb5b5d-6976-4493-a299-27c402e96ab4-kserve-provision-location\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:15.995633 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995548 2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-cabundle-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:15.995633 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995606 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "094d3dc7-e47a-49c2-892e-f1631d9eba0b" (UID: "094d3dc7-e47a-49c2-892e-f1631d9eba0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:41:15.995829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "094d3dc7-e47a-49c2-892e-f1631d9eba0b" (UID: "094d3dc7-e47a-49c2-892e-f1631d9eba0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:41:15.995829 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.995670 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72cb5b5d-6976-4493-a299-27c402e96ab4" (UID: "72cb5b5d-6976-4493-a299-27c402e96ab4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:41:15.997377 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.997348 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72cb5b5d-6976-4493-a299-27c402e96ab4" (UID: "72cb5b5d-6976-4493-a299-27c402e96ab4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:41:15.997377 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.997364 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-api-access-dlsfm" (OuterVolumeSpecName: "kube-api-access-dlsfm") pod "72cb5b5d-6976-4493-a299-27c402e96ab4" (UID: "72cb5b5d-6976-4493-a299-27c402e96ab4"). InnerVolumeSpecName "kube-api-access-dlsfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:41:15.997555 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.997533 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094d3dc7-e47a-49c2-892e-f1631d9eba0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "094d3dc7-e47a-49c2-892e-f1631d9eba0b" (UID: "094d3dc7-e47a-49c2-892e-f1631d9eba0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:41:15.997613 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:15.997596 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-api-access-vpf9f" (OuterVolumeSpecName: "kube-api-access-vpf9f") pod "094d3dc7-e47a-49c2-892e-f1631d9eba0b" (UID: "094d3dc7-e47a-49c2-892e-f1631d9eba0b"). InnerVolumeSpecName "kube-api-access-vpf9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:41:16.096461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.096381 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094d3dc7-e47a-49c2-892e-f1631d9eba0b-proxy-tls\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:16.096461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.096412 2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kserve-provision-location\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:16.096461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.096422 2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72cb5b5d-6976-4493-a299-27c402e96ab4-proxy-tls\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:16.096461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.096431 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpf9f\" (UniqueName: \"kubernetes.io/projected/094d3dc7-e47a-49c2-892e-f1631d9eba0b-kube-api-access-vpf9f\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:16.096461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.096440 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlsfm\" (UniqueName: \"kubernetes.io/projected/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-api-access-dlsfm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:16.096461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.096449 2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72cb5b5d-6976-4493-a299-27c402e96ab4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:16.096461 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.096458 2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094d3dc7-e47a-49c2-892e-f1631d9eba0b-cabundle-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 17:41:16.503391 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.503355 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" path="/var/lib/kubelet/pods/5ff1b82a-f995-47c1-9523-83ee56140b7b/volumes" Jul 27 17:41:16.722526 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.722497 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" Jul 27 17:41:16.722526 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.722505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68" event={"ID":"094d3dc7-e47a-49c2-892e-f1631d9eba0b","Type":"ContainerDied","Data":"2803e7e4a9bbb416b438c0b3369f3d432a744e53185cea064aa72f989c04b946"} Jul 27 17:41:16.722729 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.722577 2564 scope.go:117] "RemoveContainer" containerID="664e02fe69f7fea5f385065e602db3f74525a4bb99d026bbf715ecc60d629084" Jul 27 17:41:16.724576 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.724550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" event={"ID":"72cb5b5d-6976-4493-a299-27c402e96ab4","Type":"ContainerDied","Data":"b949e815ee2fee86ed7dfae605d6fa52edac2095d0f6bdb3d6a77417aa7e1fd4"} Jul 27 17:41:16.724710 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.724594 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x" Jul 27 17:41:16.731199 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.731182 2564 scope.go:117] "RemoveContainer" containerID="1e59920fabfac22f1edb1a94b2ef5d6db3043f0f99027018da3ac0a790eaddec" Jul 27 17:41:16.738945 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.738924 2564 scope.go:117] "RemoveContainer" containerID="0d461eb986c92f940230f5f4fcfd4e70bd5793c8bd13f522551288748b163af1" Jul 27 17:41:16.741815 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.741792 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68"] Jul 27 17:41:16.746133 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.746108 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-rmd68"] Jul 27 17:41:16.747596 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.747579 2564 scope.go:117] "RemoveContainer" containerID="73574124dece82b31e9a43035e7a686e6e6eee3e1b2b7262dcae2db697f578e1" Jul 27 17:41:16.754980 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.754961 2564 scope.go:117] "RemoveContainer" containerID="cfea891e9a06b4685d1fe368f936d9b55c2aeb0488387b4e1919432f04d07c10" Jul 27 17:41:16.758435 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.758411 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x"] Jul 27 17:41:16.762064 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.762044 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-hr47x"] Jul 27 17:41:16.766110 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:16.766085 2564 scope.go:117] "RemoveContainer" containerID="be919116b1d6e5790a1364b9eb88175182ee9bfa94e4a06cb57547bfbfda6fc5" Jul 27 17:41:18.504289 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:18.504255 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" path="/var/lib/kubelet/pods/094d3dc7-e47a-49c2-892e-f1631d9eba0b/volumes" Jul 27 17:41:18.504809 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:41:18.504792 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" path="/var/lib/kubelet/pods/72cb5b5d-6976-4493-a299-27c402e96ab4/volumes" Jul 27 17:42:10.822235 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:42:10.822196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:42:11.890446 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:42:11.890409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:42:42.517042 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:42:42.517008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:45:18.139439 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:45:18.139402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:45:46.882440 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:45:46.882403 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:06.407097 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:48:06.407063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:06.460127 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:48:06.460102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:12.067230 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:48:12.067197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:48:54.658951 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:48:54.658918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:25.439795 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:49:25.439760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:25.579879 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:49:25.579834 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:25.652735 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:49:25.652697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:25.729322 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:49:25.729287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:25.874901 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:49:25.874868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:25.984655 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:49:25.984552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:49:26.186228 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:49:26.186195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:25.531632 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:50:25.531594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:27.154301 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:50:27.154217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:35.840848 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:50:35.840810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:35.912740 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:50:35.912705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:35.981560 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:50:35.981526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:50:36.416446 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:50:36.416414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:25.725366 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:25.725329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:25.818327 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:25.818295 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:35.801252 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:35.801221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:35.871226 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:35.871189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:35.946016 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:35.945968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:36.015022 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:36.014993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:36.081895 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:36.081814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:36.186005 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:36.185965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:51:36.286324 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:51:36.286292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:36.041786 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:52:36.041747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:52:36.182787 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:52:36.182757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:53:20.990333 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:53:20.990299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:59:32.850562 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:59:32.850484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 17:59:44.867198 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:59:44.867161 2564 generic.go:356] "Generic (PLEG): container finished" podID="fd37b378-d9d7-403e-a97b-a6702759db69" containerID="d64432a5f115c580c5f6a10a2c889bd38397d734bec71e33c5198aa0d6d7ec60" exitCode=1 Jul 27 17:59:44.867198 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:59:44.867172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" event={"ID":"fd37b378-d9d7-403e-a97b-a6702759db69","Type":"ContainerDied","Data":"d64432a5f115c580c5f6a10a2c889bd38397d734bec71e33c5198aa0d6d7ec60"} Jul 27 17:59:44.867667 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:59:44.867523 2564 scope.go:117] "RemoveContainer" containerID="d64432a5f115c580c5f6a10a2c889bd38397d734bec71e33c5198aa0d6d7ec60" Jul 27 17:59:44.867835 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:59:44.867822 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:59:45.873254 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:59:45.873222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" event={"ID":"fd37b378-d9d7-403e-a97b-a6702759db69","Type":"ContainerStarted","Data":"1060cefe6dc66af034b0648b96171ba4acc044cc8dad0b0abbad8542db1663c2"} Jul 27 17:59:45.873653 ip-10-0-131-71 kubenswrapper[2564]: I0727 17:59:45.873443 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 18:00:16.881981 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:16.881940 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 18:00:23.649750 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:23.649717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:23.787626 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:23.787594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:23.851417 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:23.851381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:50.631715 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:50.631676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:50.749993 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:50.749964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:50.825961 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:50.825927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:50.891676 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:50.891594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:50.991294 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:50.991251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:00:51.218320 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:00:51.218246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:01:41.589766 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:01:41.589724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:01:41.652890 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:01:41.652859 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:01:50.879724 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:01:50.879686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:01:51.552743 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:01:51.552703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:05:20.704067 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:05:20.704028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:06:16.351453 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:16.351417 2564 generic.go:356] "Generic (PLEG): container finished" podID="fd37b378-d9d7-403e-a97b-a6702759db69" containerID="1060cefe6dc66af034b0648b96171ba4acc044cc8dad0b0abbad8542db1663c2" exitCode=1 Jul 27 18:06:16.351889 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:16.351491 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" event={"ID":"fd37b378-d9d7-403e-a97b-a6702759db69","Type":"ContainerDied","Data":"1060cefe6dc66af034b0648b96171ba4acc044cc8dad0b0abbad8542db1663c2"} Jul 27 18:06:16.351889 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:16.351554 2564 scope.go:117] "RemoveContainer" containerID="d64432a5f115c580c5f6a10a2c889bd38397d734bec71e33c5198aa0d6d7ec60" Jul 27 18:06:16.351889 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:16.351874 2564 scope.go:117] "RemoveContainer" containerID="1060cefe6dc66af034b0648b96171ba4acc044cc8dad0b0abbad8542db1663c2" Jul 27 18:06:16.352132 ip-10-0-131-71 kubenswrapper[2564]: E0727 18:06:16.352105 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-ff96d5959-9fj98_kserve(fd37b378-d9d7-403e-a97b-a6702759db69)\"" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" podUID="fd37b378-d9d7-403e-a97b-a6702759db69" Jul 27 18:06:16.877209 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:16.877175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 18:06:17.356847 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:17.356822 2564 scope.go:117] "RemoveContainer" containerID="1060cefe6dc66af034b0648b96171ba4acc044cc8dad0b0abbad8542db1663c2" Jul 27 18:06:17.357240 ip-10-0-131-71 kubenswrapper[2564]: E0727 18:06:17.356984 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-ff96d5959-9fj98_kserve(fd37b378-d9d7-403e-a97b-a6702759db69)\"" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" podUID="fd37b378-d9d7-403e-a97b-a6702759db69" Jul 27 18:06:19.657205 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:19.657165 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 18:06:19.657722 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:19.657661 2564 scope.go:117] "RemoveContainer" containerID="1060cefe6dc66af034b0648b96171ba4acc044cc8dad0b0abbad8542db1663c2" Jul 27 18:06:19.657903 ip-10-0-131-71 kubenswrapper[2564]: E0727 18:06:19.657878 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-ff96d5959-9fj98_kserve(fd37b378-d9d7-403e-a97b-a6702759db69)\"" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" podUID="fd37b378-d9d7-403e-a97b-a6702759db69" Jul 27 18:06:33.779116 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:33.779074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:06:34.500799 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:34.500773 2564 scope.go:117] "RemoveContainer" containerID="1060cefe6dc66af034b0648b96171ba4acc044cc8dad0b0abbad8542db1663c2" Jul 27 18:06:34.501107 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:34.501093 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 18:06:35.423658 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:35.423570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" event={"ID":"fd37b378-d9d7-403e-a97b-a6702759db69","Type":"ContainerStarted","Data":"1ef81dc81fddbf947bd437f0f997161654ad556278397c0b610bbf8309d17771"} Jul 27 18:06:35.423999 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:35.423812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 18:06:39.535984 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:39.535949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:06:48.490824 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:48.490784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:06:48.591206 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:48.591176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:06:48.998688 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:48.998652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:06:49.055528 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:06:49.055491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:07:06.433126 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:07:06.433044 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-9fj98" Jul 27 18:13:06.650775 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:13:06.650701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:15:29.841380 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:15:29.841345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:10.253037 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:10.252996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:11.680688 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680653 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slghs/must-gather-5gtkt"] Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680944 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="storage-initializer" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680955 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="storage-initializer" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680966 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="storage-initializer" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680972 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="storage-initializer" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680981 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kube-rbac-proxy" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680986 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kube-rbac-proxy" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.680995 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681000 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681008 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681013 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681020 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kube-rbac-proxy" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681027 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kube-rbac-proxy" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681045 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" Jul 27 18:18:11.681049 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681052 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681061 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="storage-initializer" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681067 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="storage-initializer" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681073 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kube-rbac-proxy" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681078 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kube-rbac-proxy" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681122 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kube-rbac-proxy" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681129 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff1b82a-f995-47c1-9523-83ee56140b7b" containerName="kserve-container" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681150 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kube-rbac-proxy" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681156 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kube-rbac-proxy" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681174 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72cb5b5d-6976-4493-a299-27c402e96ab4" containerName="kserve-container" Jul 27 18:18:11.681519 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.681180 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="094d3dc7-e47a-49c2-892e-f1631d9eba0b" containerName="kserve-container" Jul 27 18:18:11.684247 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.684227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:11.687204 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.687182 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-slghs\"/\"default-dockercfg-gm9p6\"" Jul 27 18:18:11.687341 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.687326 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slghs\"/\"openshift-service-ca.crt\"" Jul 27 18:18:11.688356 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.688342 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slghs\"/\"kube-root-ca.crt\"" Jul 27 18:18:11.691847 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.691818 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slghs/must-gather-5gtkt"] Jul 27 18:18:11.870057 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.870018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6gv\" (UniqueName: \"kubernetes.io/projected/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-kube-api-access-vf6gv\") pod \"must-gather-5gtkt\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:11.870261 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.870062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-must-gather-output\") pod \"must-gather-5gtkt\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:11.971088 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.970976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6gv\" (UniqueName: \"kubernetes.io/projected/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-kube-api-access-vf6gv\") pod \"must-gather-5gtkt\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:11.971088 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.971041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-must-gather-output\") pod \"must-gather-5gtkt\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:11.971437 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.971416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-must-gather-output\") pod \"must-gather-5gtkt\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:11.980299 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:11.980270 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6gv\" (UniqueName: \"kubernetes.io/projected/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-kube-api-access-vf6gv\") pod \"must-gather-5gtkt\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:12.004293 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:12.004258 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:12.133852 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:12.133797 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slghs/must-gather-5gtkt"] Jul 27 18:18:12.136163 ip-10-0-131-71 kubenswrapper[2564]: W0727 18:18:12.136125 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2e02407_2ad4_40c6_b9af_b8c0b8445f7f.slice/crio-100ab73d9fae663079e75915479f5e40595ac48b3086bcebe6041ee4f923387a WatchSource:0}: Error finding container 100ab73d9fae663079e75915479f5e40595ac48b3086bcebe6041ee4f923387a: Status 404 returned error can't find the container with id 100ab73d9fae663079e75915479f5e40595ac48b3086bcebe6041ee4f923387a Jul 27 18:18:12.138030 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:12.138010 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 18:18:13.040209 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:13.040171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slghs/must-gather-5gtkt" event={"ID":"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f","Type":"ContainerStarted","Data":"100ab73d9fae663079e75915479f5e40595ac48b3086bcebe6041ee4f923387a"} Jul 27 18:18:14.072593 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:14.072530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:19.067544 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:19.067503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slghs/must-gather-5gtkt" event={"ID":"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f","Type":"ContainerStarted","Data":"8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6"} Jul 27 18:18:19.067544 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:19.067553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slghs/must-gather-5gtkt" event={"ID":"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f","Type":"ContainerStarted","Data":"b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825"} Jul 27 18:18:19.083731 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:19.083661 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slghs/must-gather-5gtkt" podStartSLOduration=1.9369872620000002 podStartE2EDuration="8.083629167s" totalImagesPullingTime="6.146641905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:19.083449956 +0000 UTC m=+3015.082235778" watchObservedRunningTime="2026-07-27 18:18:19.083629167 +0000 UTC m=+3015.082414988" Jul 27 18:18:42.171247 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:42.171209 2564 generic.go:356] "Generic (PLEG): container finished" podID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerID="b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825" exitCode=0 Jul 27 18:18:42.171703 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:42.171284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slghs/must-gather-5gtkt" event={"ID":"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f","Type":"ContainerDied","Data":"b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825"} Jul 27 18:18:42.171703 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:42.171604 2564 scope.go:117] "RemoveContainer" containerID="b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825" Jul 27 18:18:42.682958 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:42.682923 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-slghs_must-gather-5gtkt_a2e02407-2ad4-40c6-b9af-b8c0b8445f7f/gather/0.log" Jul 27 18:18:43.131555 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.131516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:43.288442 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.288397 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mlzzr/must-gather-qvdvh"] Jul 27 18:18:43.292283 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.292256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.294620 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.294583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:43.295423 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.295403 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlzzr\"/\"kube-root-ca.crt\"" Jul 27 18:18:43.295535 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.295403 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mlzzr\"/\"default-dockercfg-hfqqn\"" Jul 27 18:18:43.295535 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.295480 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlzzr\"/\"openshift-service-ca.crt\"" Jul 27 18:18:43.298917 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.298895 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlzzr/must-gather-qvdvh"] Jul 27 18:18:43.346274 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.346232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsff2\" (UniqueName: \"kubernetes.io/projected/c7a133ff-a837-4a5e-955a-d99089fb0101-kube-api-access-nsff2\") pod \"must-gather-qvdvh\" (UID: \"c7a133ff-a837-4a5e-955a-d99089fb0101\") " pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.346274 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.346277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7a133ff-a837-4a5e-955a-d99089fb0101-must-gather-output\") pod \"must-gather-qvdvh\" (UID: \"c7a133ff-a837-4a5e-955a-d99089fb0101\") " pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.395423 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.395339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:43.447409 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.447366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsff2\" (UniqueName: \"kubernetes.io/projected/c7a133ff-a837-4a5e-955a-d99089fb0101-kube-api-access-nsff2\") pod \"must-gather-qvdvh\" (UID: \"c7a133ff-a837-4a5e-955a-d99089fb0101\") " pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.447409 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.447412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7a133ff-a837-4a5e-955a-d99089fb0101-must-gather-output\") pod \"must-gather-qvdvh\" (UID: \"c7a133ff-a837-4a5e-955a-d99089fb0101\") " pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.447760 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.447743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c7a133ff-a837-4a5e-955a-d99089fb0101-must-gather-output\") pod \"must-gather-qvdvh\" (UID: \"c7a133ff-a837-4a5e-955a-d99089fb0101\") " pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.456024 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.455991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsff2\" (UniqueName: \"kubernetes.io/projected/c7a133ff-a837-4a5e-955a-d99089fb0101-kube-api-access-nsff2\") pod \"must-gather-qvdvh\" (UID: \"c7a133ff-a837-4a5e-955a-d99089fb0101\") " pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.603567 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.603532 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlzzr/must-gather-qvdvh" Jul 27 18:18:43.731800 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:43.731764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlzzr/must-gather-qvdvh"] Jul 27 18:18:43.734763 ip-10-0-131-71 kubenswrapper[2564]: W0727 18:18:43.734734 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7a133ff_a837_4a5e_955a_d99089fb0101.slice/crio-5a74238171a1cd67047f22790093cd3d119c2fc7bbec5310f152ef38a458ac52 WatchSource:0}: Error finding container 5a74238171a1cd67047f22790093cd3d119c2fc7bbec5310f152ef38a458ac52: Status 404 returned error can't find the container with id 5a74238171a1cd67047f22790093cd3d119c2fc7bbec5310f152ef38a458ac52 Jul 27 18:18:44.179920 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:44.179884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlzzr/must-gather-qvdvh" event={"ID":"c7a133ff-a837-4a5e-955a-d99089fb0101","Type":"ContainerStarted","Data":"5a74238171a1cd67047f22790093cd3d119c2fc7bbec5310f152ef38a458ac52"} Jul 27 18:18:45.186962 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:45.186926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlzzr/must-gather-qvdvh" event={"ID":"c7a133ff-a837-4a5e-955a-d99089fb0101","Type":"ContainerStarted","Data":"c2d98e590edc6ca58d33d13c91a207c2514bb94f3e877a78113d3bf1d76f167e"} Jul 27 18:18:46.195945 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:46.195883 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlzzr/must-gather-qvdvh" event={"ID":"c7a133ff-a837-4a5e-955a-d99089fb0101","Type":"ContainerStarted","Data":"0ae603b71a9c6abcba15e2aa5d1906780f65c79a7f2cd79be43759e78818da5b"} Jul 27 18:18:46.214373 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:46.214321 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mlzzr/must-gather-qvdvh" podStartSLOduration=1.952847714 podStartE2EDuration="3.214300514s" totalImagesPullingTime="1.2614528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:46.212607204 +0000 UTC m=+3042.211393025" watchObservedRunningTime="2026-07-27 18:18:46.214300514 +0000 UTC m=+3042.213086335" Jul 27 18:18:46.399604 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:46.399543 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2cq5f_b53269e6-9bf1-4b56-8c80-73a3e3f630dd/global-pull-secret-syncer/0.log" Jul 27 18:18:46.575642 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:46.575610 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" Jul 27 18:18:46.696695 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:46.696664 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-71.ec2.internal_3498390b3a0f07e184eb7f66c8538395/haproxy/0.log" Jul 27 18:18:48.118184 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.118149 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-slghs/must-gather-5gtkt"] Jul 27 18:18:48.121507 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.121475 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-slghs/must-gather-5gtkt"] Jul 27 18:18:48.122957 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.122543 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-slghs/must-gather-5gtkt" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerName="copy" containerID="cri-o://8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6" gracePeriod=2 Jul 27 18:18:48.124931 ip-10-0-131-71 kubenswrapper[2564]: E0727 18:18:48.124905 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5gtkt\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-slghs\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:48.484178 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.483613 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-slghs_must-gather-5gtkt_a2e02407-2ad4-40c6-b9af-b8c0b8445f7f/copy/0.log" Jul 27 18:18:48.489161 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.485918 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:48.492164 ip-10-0-131-71 kubenswrapper[2564]: E0727 18:18:48.490241 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5gtkt\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-slghs\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:48.604054 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.603407 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-must-gather-output\") pod \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " Jul 27 18:18:48.604054 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.603514 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf6gv\" (UniqueName: \"kubernetes.io/projected/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-kube-api-access-vf6gv\") pod \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\" (UID: \"a2e02407-2ad4-40c6-b9af-b8c0b8445f7f\") " Jul 27 18:18:48.607071 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.605969 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" (UID: "a2e02407-2ad4-40c6-b9af-b8c0b8445f7f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 18:18:48.607392 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.607352 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-kube-api-access-vf6gv" (OuterVolumeSpecName: "kube-api-access-vf6gv") pod "a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" (UID: "a2e02407-2ad4-40c6-b9af-b8c0b8445f7f"). InnerVolumeSpecName "kube-api-access-vf6gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 18:18:48.705030 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.704917 2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-must-gather-output\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 18:18:48.705030 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.704952 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6gv\" (UniqueName: \"kubernetes.io/projected/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f-kube-api-access-vf6gv\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 27 18:18:48.899102 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.898931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:48.999590 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:48.999419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zv6d_b8077354-83d5-412f-9b21-3b4878a000c5/konnectivity-agent/0.log" limit=1024 Jul 27 18:18:49.218370 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.218343 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-slghs_must-gather-5gtkt_a2e02407-2ad4-40c6-b9af-b8c0b8445f7f/copy/0.log" Jul 27 18:18:49.219338 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.219309 2564 generic.go:356] "Generic (PLEG): container finished" podID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerID="8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6" exitCode=143 Jul 27 18:18:49.219904 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.219720 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slghs/must-gather-5gtkt" Jul 27 18:18:49.225921 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.219756 2564 scope.go:117] "RemoveContainer" containerID="8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6" Jul 27 18:18:49.248499 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.248473 2564 scope.go:117] "RemoveContainer" containerID="b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825" Jul 27 18:18:49.287407 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.287372 2564 scope.go:117] "RemoveContainer" containerID="8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6" Jul 27 18:18:49.287925 ip-10-0-131-71 kubenswrapper[2564]: E0727 18:18:49.287887 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6\": container with ID starting with 8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6 not found: ID does not exist" containerID="8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6" Jul 27 18:18:49.288048 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.287933 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6"} err="failed to get container status \"8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6\": rpc error: code = NotFound desc = could not find container \"8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6\": container with ID starting with 8127fb31fd19a2f2e699a235df916760d8a21d34ead9b911b2cc1c97305cbef6 not found: ID does not exist" Jul 27 18:18:49.288048 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.287960 2564 scope.go:117] "RemoveContainer" containerID="b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825" Jul 27 18:18:49.288485 ip-10-0-131-71 kubenswrapper[2564]: E0727 18:18:49.288387 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825\": container with ID starting with b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825 not found: ID does not exist" containerID="b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825" Jul 27 18:18:49.288485 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:49.288421 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825"} err="failed to get container status \"b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825\": rpc error: code = NotFound desc = could not find container \"b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825\": container with ID starting with b567860b7d138e187ddebc0c285b8120f8509cce840aa772e59d23a040c27825 not found: ID does not exist" Jul 27 18:18:50.507722 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:50.507668 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" path="/var/lib/kubelet/pods/a2e02407-2ad4-40c6-b9af-b8c0b8445f7f/volumes" Jul 27 18:18:51.471320 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.471205 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mcrs9_5c29a54b-c791-496a-8947-6e68af57a0e8/node-exporter/0.log" Jul 27 18:18:51.493961 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.493930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mcrs9_5c29a54b-c791-496a-8947-6e68af57a0e8/kube-rbac-proxy/0.log" Jul 27 18:18:51.522608 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.522580 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mcrs9_5c29a54b-c791-496a-8947-6e68af57a0e8/init-textfile/0.log" Jul 27 18:18:51.670018 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.669986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0bd78e95-4c46-4239-a83b-80112fabfbc4/prometheus/0.log" Jul 27 18:18:51.689126 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.689076 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0bd78e95-4c46-4239-a83b-80112fabfbc4/config-reloader/0.log" Jul 27 18:18:51.713124 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.713094 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0bd78e95-4c46-4239-a83b-80112fabfbc4/thanos-sidecar/0.log" Jul 27 18:18:51.738456 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.738415 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0bd78e95-4c46-4239-a83b-80112fabfbc4/kube-rbac-proxy-web/0.log" Jul 27 18:18:51.761569 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.761529 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0bd78e95-4c46-4239-a83b-80112fabfbc4/kube-rbac-proxy/0.log" Jul 27 18:18:51.785119 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.785085 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0bd78e95-4c46-4239-a83b-80112fabfbc4/kube-rbac-proxy-thanos/0.log" Jul 27 18:18:51.807908 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.807876 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0bd78e95-4c46-4239-a83b-80112fabfbc4/init-config-reloader/0.log" Jul 27 18:18:51.840583 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.840546 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-drtbp_176d2221-4acd-483e-9630-bebac5a413e8/prometheus-operator/0.log" Jul 27 18:18:51.857924 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:51.857890 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-drtbp_176d2221-4acd-483e-9630-bebac5a413e8/kube-rbac-proxy/0.log" Jul 27 18:18:52.023675 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:52.023571 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b8b8448-gw5vz_5d14aa51-1b66-4d61-9515-61edbf06d2cb/thanos-query/0.log" Jul 27 18:18:52.045732 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:52.045685 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b8b8448-gw5vz_5d14aa51-1b66-4d61-9515-61edbf06d2cb/kube-rbac-proxy-web/0.log" Jul 27 18:18:52.071080 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:52.071052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b8b8448-gw5vz_5d14aa51-1b66-4d61-9515-61edbf06d2cb/kube-rbac-proxy/0.log" Jul 27 18:18:52.093409 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:52.093378 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b8b8448-gw5vz_5d14aa51-1b66-4d61-9515-61edbf06d2cb/prom-label-proxy/0.log" Jul 27 18:18:52.120603 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:52.120571 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b8b8448-gw5vz_5d14aa51-1b66-4d61-9515-61edbf06d2cb/kube-rbac-proxy-rules/0.log" Jul 27 18:18:52.145945 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:52.145913 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b8b8448-gw5vz_5d14aa51-1b66-4d61-9515-61edbf06d2cb/kube-rbac-proxy-metrics/0.log" Jul 27 18:18:54.436127 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:54.436095 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c95477bc9-l7fk8_ab2f07c2-8828-4767-a63e-2aa4928ffe2f/console/0.log" Jul 27 18:18:55.069475 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.069429 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd"] Jul 27 18:18:55.069986 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.069960 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerName="gather" Jul 27 18:18:55.070102 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.069989 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerName="gather" Jul 27 18:18:55.070102 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.070012 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerName="copy" Jul 27 18:18:55.070102 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.070020 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerName="copy" Jul 27 18:18:55.070102 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.070092 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerName="copy" Jul 27 18:18:55.070344 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.070109 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e02407-2ad4-40c6-b9af-b8c0b8445f7f" containerName="gather" Jul 27 18:18:55.074188 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.074160 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.080788 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.080754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd"] Jul 27 18:18:55.170041 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.169996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-lib-modules\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.170041 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.170046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-proc\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.170318 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.170073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-podres\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.170318 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.170208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-sys\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.170318 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.170242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97cbq\" (UniqueName: \"kubernetes.io/projected/87d5362a-0065-40f3-83d8-e74612204755-kube-api-access-97cbq\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.270802 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.270757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-lib-modules\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.270802 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.270809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-proc\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.271055 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.270837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-podres\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.271055 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.270954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-sys\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.271055 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.270962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-lib-modules\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.271055 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.270995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97cbq\" (UniqueName: \"kubernetes.io/projected/87d5362a-0065-40f3-83d8-e74612204755-kube-api-access-97cbq\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.271055 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.270961 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-proc\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.271325 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.271083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-podres\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.271325 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.271092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87d5362a-0065-40f3-83d8-e74612204755-sys\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.280217 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.280184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97cbq\" (UniqueName: \"kubernetes.io/projected/87d5362a-0065-40f3-83d8-e74612204755-kube-api-access-97cbq\") pod \"perf-node-gather-daemonset-p7jcd\" (UID: \"87d5362a-0065-40f3-83d8-e74612204755\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.388886 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.388796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:55.562154 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:55.561667 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd"] Jul 27 18:18:55.565391 ip-10-0-131-71 kubenswrapper[2564]: W0727 18:18:55.565356 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87d5362a_0065_40f3_83d8_e74612204755.slice/crio-53af139d29edb8f8b2bc7eda1e308d0fdd1811516a6a6094a23332a1e5fd35e0 WatchSource:0}: Error finding container 53af139d29edb8f8b2bc7eda1e308d0fdd1811516a6a6094a23332a1e5fd35e0: Status 404 returned error can't find the container with id 53af139d29edb8f8b2bc7eda1e308d0fdd1811516a6a6094a23332a1e5fd35e0 Jul 27 18:18:56.018924 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.018889 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w2hvc_408b8bb0-3549-4640-b502-6ab8c6c97fec/dns/0.log" Jul 27 18:18:56.039671 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.039639 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w2hvc_408b8bb0-3549-4640-b502-6ab8c6c97fec/kube-rbac-proxy/0.log" Jul 27 18:18:56.063086 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.063062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dsrdc_6d3767a9-6ba7-481d-b6bf-b2f76eb5a679/dns-node-resolver/0.log" Jul 27 18:18:56.266872 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.266830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" event={"ID":"87d5362a-0065-40f3-83d8-e74612204755","Type":"ContainerStarted","Data":"e2a5ee54de0aa4e8d8660479b059c92a352ef1389f540fbfee0110f75461348e"} Jul 27 18:18:56.267072 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.266896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" event={"ID":"87d5362a-0065-40f3-83d8-e74612204755","Type":"ContainerStarted","Data":"53af139d29edb8f8b2bc7eda1e308d0fdd1811516a6a6094a23332a1e5fd35e0"} Jul 27 18:18:56.267072 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.266959 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:18:56.284651 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.284544 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" podStartSLOduration=1.284527301 podStartE2EDuration="1.284527301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:56.282603455 +0000 UTC m=+3052.281389276" watchObservedRunningTime="2026-07-27 18:18:56.284527301 +0000 UTC m=+3052.283313125" Jul 27 18:18:56.673856 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:56.673756 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvbv6_4be3196f-ed07-462b-a955-e78df488a631/node-ca/0.log" Jul 27 18:18:57.762502 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:57.762466 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xmskc_4354227a-bd56-4d80-b591-fa7e964b1759/serve-healthcheck-canary/0.log" Jul 27 18:18:58.146516 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:58.146446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-75nxl_6756dc88-971f-4e21-b2d9-aac9389f8b74/kube-rbac-proxy/0.log" Jul 27 18:18:58.167661 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:58.167631 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-75nxl_6756dc88-971f-4e21-b2d9-aac9389f8b74/exporter/0.log" Jul 27 18:18:58.189917 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:18:58.189880 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-75nxl_6756dc88-971f-4e21-b2d9-aac9389f8b74/extractor/0.log" Jul 27 18:19:00.458944 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:00.458899 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-9fj98_fd37b378-d9d7-403e-a97b-a6702759db69/manager/2.log" Jul 27 18:19:00.468408 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:00.468372 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-9fj98_fd37b378-d9d7-403e-a97b-a6702759db69/manager/1.log" Jul 27 18:19:00.687709 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:00.687671 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-tjfjs_2def4a9c-07d8-4703-910d-3e48801c3b2b/manager/0.log" Jul 27 18:19:00.730232 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:00.730199 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-svwhx_1712a34c-40a4-40cc-841a-e2453c7028b2/s3-init/0.log" Jul 27 18:19:00.747107 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:00.747082 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-svwhx_1712a34c-40a4-40cc-841a-e2453c7028b2/download-hf-model/0.log" Jul 27 18:19:00.779213 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:00.779183 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-g2kfm_1da230c8-be4b-4648-8138-ec6e80d1fbf1/seaweedfs/0.log" Jul 27 18:19:02.283376 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:02.283346 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-p7jcd" Jul 27 18:19:04.849545 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:04.849506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-7w6rk_da828fde-51c6-4ce5-b59a-a40dafb28655/migrator/0.log" Jul 27 18:19:04.875468 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:04.875438 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-7w6rk_da828fde-51c6-4ce5-b59a-a40dafb28655/graceful-termination/0.log" Jul 27 18:19:06.428380 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.428310 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xkvfz_74acacfd-72eb-4498-9437-385f6d5d5ec2/kube-multus-additional-cni-plugins/0.log" Jul 27 18:19:06.450335 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.450309 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xkvfz_74acacfd-72eb-4498-9437-385f6d5d5ec2/egress-router-binary-copy/0.log" Jul 27 18:19:06.476024 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.475993 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xkvfz_74acacfd-72eb-4498-9437-385f6d5d5ec2/cni-plugins/0.log" Jul 27 18:19:06.500242 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.500211 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xkvfz_74acacfd-72eb-4498-9437-385f6d5d5ec2/bond-cni-plugin/0.log" Jul 27 18:19:06.522046 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.522002 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xkvfz_74acacfd-72eb-4498-9437-385f6d5d5ec2/routeoverride-cni/0.log" Jul 27 18:19:06.546405 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.546377 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xkvfz_74acacfd-72eb-4498-9437-385f6d5d5ec2/whereabouts-cni-bincopy/0.log" Jul 27 18:19:06.568429 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.568402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xkvfz_74acacfd-72eb-4498-9437-385f6d5d5ec2/whereabouts-cni/0.log" Jul 27 18:19:06.604221 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.604193 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gslm6_c3ecd635-1afa-47cb-9dd1-e43b956e43a6/kube-multus/0.log" Jul 27 18:19:06.704448 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.704362 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6gcqw_861d178f-ae27-4311-83a0-e0e3a9046297/network-metrics-daemon/0.log" Jul 27 18:19:06.723353 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:06.723319 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6gcqw_861d178f-ae27-4311-83a0-e0e3a9046297/kube-rbac-proxy/0.log" Jul 27 18:19:07.901012 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:07.900978 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/ovn-controller/0.log" Jul 27 18:19:07.916753 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:07.916728 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/ovn-acl-logging/0.log" Jul 27 18:19:07.939740 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:07.939707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/kube-rbac-proxy-node/0.log" Jul 27 18:19:07.963084 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:07.963059 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/kube-rbac-proxy-ovn-metrics/0.log" Jul 27 18:19:07.981796 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:07.981772 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/northd/0.log" Jul 27 18:19:08.001093 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:08.001072 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/nbdb/0.log" Jul 27 18:19:08.020671 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:08.020651 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/sbdb/0.log" Jul 27 18:19:08.085260 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:08.085230 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r948w_fd97aaac-23af-44f8-865e-6cfeff89df73/ovnkube-controller/0.log" Jul 27 18:19:09.102858 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:09.102829 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-w9rd7_736e470f-079a-4aea-b75c-c4ac3cf684bf/network-check-target-container/0.log" Jul 27 18:19:10.086584 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:10.086551 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zndfc_027bfb55-a6e1-4a7c-ba3a-59e4adaa97fd/iptables-alerter/0.log" Jul 27 18:19:10.658560 ip-10-0-131-71 kubenswrapper[2564]: I0727 18:19:10.658523 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9cz8t_3d70e7cf-9907-43e7-96e9-b7b421bbaffa/tuned/0.log"