Jul 07 02:17:43.503232 ip-10-0-138-74 systemd[1]: Starting Kubernetes Kubelet... Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.967815 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973796 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973809 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973813 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973816 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973821 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973824 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973833 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:44.033274 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973836 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973838 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973841 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973844 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973847 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973849 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973852 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973854 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973857 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973859 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973862 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973864 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973869 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973872 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973875 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973878 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973881 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973884 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973888 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973892 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:44.047906 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973895 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973898 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973901 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973905 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973907 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973910 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973912 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973915 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973918 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973920 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973923 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973926 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973929 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973932 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973935 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973938 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973940 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973943 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973946 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973948 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:44.049983 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973951 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:44.050271 ip-10-0-138-74 systemd[1]: Started Kubernetes Kubelet. Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973953 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973956 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973959 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973961 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973964 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973969 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973972 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973975 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973978 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973981 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973984 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973987 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973989 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973992 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973995 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.973997 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974000 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974002 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:44.052821 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974005 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974010 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974013 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974015 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974018 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974021 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974024 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974027 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974029 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974032 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974034 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974037 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974040 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974042 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974045 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974047 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974050 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974052 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974055 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974057 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:44.054257 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974060 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974062 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974065 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974068 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974070 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974073 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974076 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974078 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974081 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974083 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974086 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974088 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974091 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974810 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974818 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974821 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974824 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974827 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974830 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:44.058507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974833 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974836 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974839 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974842 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974844 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974847 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974849 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974852 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974854 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974857 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974859 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974863 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974868 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974870 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974873 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974877 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974880 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974884 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974886 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974889 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:44.060325 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974891 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974894 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974896 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974899 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974901 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974904 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974906 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974909 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974912 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974914 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974917 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974920 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974923 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974925 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974929 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974931 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974934 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974936 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974939 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974941 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:44.063137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974944 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974947 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974949 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974952 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974955 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974957 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974962 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974965 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974968 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974971 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974974 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974977 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974980 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974982 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974985 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974988 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974991 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974993 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974996 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.974998 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:44.065344 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975001 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975004 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975007 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975010 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975013 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975015 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975018 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975021 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975024 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975026 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975029 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975032 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975034 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975037 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975039 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975042 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975045 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975047 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975050 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975052 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:44.067919 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975055 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975057 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975060 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975066 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975069 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975072 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975074 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975077 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975080 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975082 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975085 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975089 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975092 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975469 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975475 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975478 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975481 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975486 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975490 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:44.070107 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975493 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975499 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975502 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975505 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975508 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975511 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975513 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975516 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975518 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975521 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975523 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975526 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975528 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975531 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975533 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975536 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975540 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975543 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975546 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:44.073952 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975548 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975551 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975554 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975556 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975559 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975562 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975564 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975567 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975569 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975572 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975574 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975577 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975580 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975582 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975588 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975590 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975593 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975597 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975601 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975603 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:44.087662 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975606 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975608 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975611 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975613 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975615 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975618 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975620 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975622 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975625 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975628 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975631 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975633 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975636 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975638 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975641 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975644 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975646 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975649 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975651 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975654 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:44.090507 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975656 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975659 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975661 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975663 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975666 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975669 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975672 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975675 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975677 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975680 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975682 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975684 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975687 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975689 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975691 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975694 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975708 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975711 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975713 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975716 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:44.240251 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975718 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975722 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975724 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975727 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975730 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975732 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975735 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975737 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975740 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975744 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975747 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975751 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975754 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.975757 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975824 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975832 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975837 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975842 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975848 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975851 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975856 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 02:17:44.259014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975860 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975863 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975866 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975869 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975872 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975876 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975879 2557 flags.go:64] FLAG: --cgroup-root="" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975882 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975884 2557 flags.go:64] FLAG: --client-ca-file="" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975887 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975890 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975894 2557 flags.go:64] FLAG: --cluster-domain="" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975898 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975901 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975904 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975908 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975911 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975914 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975917 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975920 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975923 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975926 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975930 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975933 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 02:17:44.261114 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975936 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975940 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975942 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975945 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975948 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975953 2557 flags.go:64] FLAG: --enable-server="true" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975956 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975960 2557 flags.go:64] FLAG: --event-burst="100" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975964 2557 flags.go:64] FLAG: --event-qps="50" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975967 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975970 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975973 2557 flags.go:64] FLAG: --eviction-hard="" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975981 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975984 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975987 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975990 2557 flags.go:64] FLAG: --eviction-soft="" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975993 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975996 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.975999 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976002 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976005 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976008 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976011 2557 flags.go:64] FLAG: --feature-gates="" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976015 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976018 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 02:17:44.263793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976021 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976024 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976027 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976030 2557 flags.go:64] FLAG: --help="false" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976032 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976035 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976038 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976041 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976044 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976048 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976050 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976053 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976057 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976060 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976063 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976066 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976069 2557 flags.go:64] FLAG: --kube-reserved="" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976072 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976075 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976077 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976080 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976083 2557 flags.go:64] FLAG: --lock-file="" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976086 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976089 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 02:17:44.265784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976091 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976096 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976099 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976103 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976106 2557 flags.go:64] FLAG: --logging-format="text" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976108 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976112 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976115 2557 flags.go:64] FLAG: --manifest-url="" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976118 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976122 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976125 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976129 2557 flags.go:64] FLAG: --max-pods="110" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976131 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976134 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976137 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976140 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976143 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976146 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976149 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976156 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976160 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976163 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976166 2557 flags.go:64] FLAG: --pod-cidr="" Jul 07 02:17:44.266541 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976170 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976175 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976178 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976181 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976184 2557 flags.go:64] FLAG: --port="10250" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976187 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976190 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f340ffe89c8d4d60" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976193 2557 flags.go:64] FLAG: --qos-reserved="" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976196 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976199 2557 flags.go:64] FLAG: --register-node="true" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976201 2557 flags.go:64] FLAG: --register-with-taints="" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976205 2557 flags.go:64] FLAG: --registry-burst="10" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976207 2557 flags.go:64] FLAG: --registry-qps="5" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976211 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976214 2557 flags.go:64] FLAG: --reserved-memory="" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976218 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976220 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976224 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976227 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976230 2557 flags.go:64] FLAG: --runonce="false" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976233 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976235 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976238 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976241 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976244 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976248 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 02:17:44.272652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976250 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976253 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976256 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976260 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976262 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976265 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976268 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976271 2557 flags.go:64] FLAG: --system-cgroups="" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976274 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976279 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976282 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976285 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976289 2557 flags.go:64] FLAG: --tls-min-version="" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976292 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976295 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976297 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976300 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976303 2557 flags.go:64] FLAG: --v="2" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976307 2557 flags.go:64] FLAG: --version="false" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976312 2557 flags.go:64] FLAG: --vmodule="" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976315 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976318 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976447 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976453 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:44.273455 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976456 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976459 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976462 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976465 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976468 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976471 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976474 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976476 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976479 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976481 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976483 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976488 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976490 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976493 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976495 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976497 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976500 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976504 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976508 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976510 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:44.275360 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976513 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976515 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976518 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976520 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976523 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976525 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976528 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976532 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976534 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976537 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976539 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976542 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976544 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976546 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976549 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976552 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976554 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976556 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976559 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976561 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:44.275979 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976564 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976566 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976570 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976574 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976577 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976579 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976583 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976586 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976589 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976592 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976595 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976599 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976602 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976604 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976607 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976610 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976613 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976615 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976618 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976623 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:44.276580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976626 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976628 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976631 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976634 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976636 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976639 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976641 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976644 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976646 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976649 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976651 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976654 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976656 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976659 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976661 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976665 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976668 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976670 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976672 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976675 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:44.277188 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976678 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976680 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976683 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976685 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976688 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976690 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976693 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976695 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976698 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976700 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976703 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976706 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976709 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976712 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976714 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976717 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:44.277729 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.976719 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.976725 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.986604 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.986623 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.986645 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.986651 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986721 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986727 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986730 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986733 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986736 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986740 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986743 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986745 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986748 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:44.553336 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986751 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986754 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986757 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986759 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986762 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986765 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986767 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986770 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986772 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986775 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986777 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986780 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986782 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986786 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986788 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986792 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986796 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986800 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986804 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:44.553927 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986807 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986813 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986815 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986818 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986821 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986823 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986826 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986829 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986831 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986834 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986837 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986841 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986844 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986847 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986851 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986854 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986857 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986860 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986862 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:44.554586 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986865 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986869 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986871 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986874 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986876 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986879 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986881 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986884 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986886 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986890 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986894 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986896 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986899 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986901 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986906 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986909 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986912 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986915 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986918 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986921 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:44.555108 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986924 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986927 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986929 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986932 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986935 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986937 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986940 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986944 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986947 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986949 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986952 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986955 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986958 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986960 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986963 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986966 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986968 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986971 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986973 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986976 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:44.555710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986978 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986981 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986983 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986986 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986989 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986991 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986995 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.986998 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987001 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987003 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987006 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987008 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.987014 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987152 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987157 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:44.556243 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987160 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987163 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987166 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987169 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987173 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987176 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987179 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987181 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987184 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987187 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987190 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987192 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987195 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987198 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987201 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987203 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987207 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987210 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987213 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987216 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:44.556710 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987219 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987221 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987224 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987227 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987231 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987233 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987236 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987238 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987241 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987243 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987246 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987249 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987252 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987254 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987257 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987259 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987264 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987267 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987269 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:44.557246 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987272 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987274 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987277 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987280 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987286 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987288 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987291 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987293 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987296 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987298 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987301 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987305 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987308 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987311 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987314 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987317 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987319 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987323 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987326 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:44.557810 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987329 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987332 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987334 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987337 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987340 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987342 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987345 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987349 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987352 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987355 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987360 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987363 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987365 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987368 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987384 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987387 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987390 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987393 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987396 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:44.558311 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987399 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987402 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987404 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987407 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987410 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987412 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987415 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987417 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987420 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987423 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987425 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987429 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987432 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987435 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987437 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987440 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987442 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987445 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987448 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:44.558881 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:43.987451 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.987456 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.987584 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.991780 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.993385 2557 server.go:1019] "Starting client certificate rotation" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.993479 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.993520 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:43.999235 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.000933 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.012578 2557 log.go:25] "Validated CRI v1 runtime API" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.016942 2557 log.go:25] "Validated CRI v1 image API" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.018728 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 02:17:44.559399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.021225 2557 fs.go:135] Filesystem UUIDs: map[1573f325-f956-4809-a888-ea37791d63a8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 852847a6-2244-4bc5-81de-7a60ec30495c:/dev/nvme0n1p3] Jul 07 02:17:44.559798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.021241 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 02:17:44.559798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.028166 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.028447 2557 manager.go:216] Machine: {Timestamp:2026-07-07 02:17:44.026519205 +0000 UTC m=+0.389721783 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105165 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f83d1ea285c63f04f8dbb5801c9e7 SystemUUID:ec2f83d1-ea28-5c63-f04f-8dbb5801c9e7 BootID:10b01776-06a8-47a2-9c11-8d4e2651f054 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6d:60:32:ee:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6d:60:32:ee:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:dd:84:45:63:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.028556 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.028651 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.031714 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.031740 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.032437 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.032448 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.032456 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.033209 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.034529 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.037918 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038031 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038058 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038068 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038085 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038094 2557 kubelet.go:398] "Adding apiserver pod source" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038107 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 02:17:44.559975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038438 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.038457 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.041197 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.043438 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.043466 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044585 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044600 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044610 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044620 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044630 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044635 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044644 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044649 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044657 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044662 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044671 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.044676 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.045326 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.045333 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.048492 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.048493 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.048517 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.049226 2557 server.go:1268] "Started kubelet" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.049328 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.049454 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.049517 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.051240 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.051475 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.054420 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.056917 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-74.ec2.internal.18bfe05c3266662a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-74.ec2.internal,UID:ip-10-0-138-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-74.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.049190442 +0000 UTC m=+0.412393017,LastTimestamp:2026-07-07 02:17:44.049190442 +0000 UTC m=+0.412393017,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-74.ec2.internal,}" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.060508 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.060528 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.060815 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061273 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061450 2557 factory.go:55] Registering systemd factory Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061484 2557 factory.go:223] Registration of the systemd container factory successfully Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061542 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061553 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061717 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061725 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061761 2557 factory.go:153] Registering CRI-O factory Jul 07 02:17:44.560929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061781 2557 factory.go:223] Registration of the crio container factory successfully Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061840 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061879 2557 factory.go:103] Registering Raw factory Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.061899 2557 manager.go:1196] Started watching for new ooms in manager Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.062466 2557 manager.go:349] Starting recovery of all containers Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.073510 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.073613 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.078497 2557 manager.go:354] Recovery completed Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.080307 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrzvd" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.082879 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.084983 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.085011 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.085021 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.085546 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.085557 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.085574 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.085559 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.086927 2557 policy_none.go:49] "None policy: Start" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.086945 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.086957 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.088611 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrzvd" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.088936 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-74.ec2.internal.18bfe05c3488c451 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-74.ec2.internal,UID:ip-10-0-138-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-74.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-74.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.084997201 +0000 UTC m=+0.448199775,LastTimestamp:2026-07-07 02:17:44.084997201 +0000 UTC m=+0.448199775,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-74.ec2.internal,}" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.089429 2557 policy_none.go:47] "Start" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.090951 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.090977 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.090992 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.133142 2557 manager.go:337] "Starting Device Plugin manager" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.133395 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.133409 2557 server.go:87] "Starting device plugin registration server" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.133678 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.133689 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.133795 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.133881 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.133889 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.134318 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.134366 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.192630 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.193881 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.193929 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 02:17:44.562072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.193957 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.193998 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.196667 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.234427 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.235511 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.235544 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.235561 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.235590 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.244262 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.244285 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-74.ec2.internal\": node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.274239 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.294982 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal"] Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.295047 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.296772 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.296805 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.296816 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298214 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298339 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298368 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298951 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298971 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298983 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298990 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.298997 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.299004 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.300282 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.300305 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.300970 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.300997 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.301006 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.333202 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-74.ec2.internal\" not found" node="ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.339743 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-74.ec2.internal\" not found" node="ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.362970 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc0889d5315af6f41b8c9a541959d87a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"bc0889d5315af6f41b8c9a541959d87a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.362993 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0889d5315af6f41b8c9a541959d87a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"bc0889d5315af6f41b8c9a541959d87a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.838588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.363010 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe7fbc4b9b8b263ad709e9ac1b28ed97-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"fe7fbc4b9b8b263ad709e9ac1b28ed97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.375184 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.463754 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0889d5315af6f41b8c9a541959d87a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"bc0889d5315af6f41b8c9a541959d87a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.463785 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe7fbc4b9b8b263ad709e9ac1b28ed97-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"fe7fbc4b9b8b263ad709e9ac1b28ed97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.463804 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc0889d5315af6f41b8c9a541959d87a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"bc0889d5315af6f41b8c9a541959d87a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.463849 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0889d5315af6f41b8c9a541959d87a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"bc0889d5315af6f41b8c9a541959d87a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.463864 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fe7fbc4b9b8b263ad709e9ac1b28ed97-config\") pod \"kube-apiserver-proxy-ip-10-0-138-74.ec2.internal\" (UID: \"fe7fbc4b9b8b263ad709e9ac1b28ed97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.463856 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc0889d5315af6f41b8c9a541959d87a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal\" (UID: \"bc0889d5315af6f41b8c9a541959d87a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.475829 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.576623 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.637804 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.643018 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.677434 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:44.840679 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.777938 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:44.878104 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-74.ec2.internal\" not found" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.918289 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.963739 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.981116 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.984025 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.991590 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.992736 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.992848 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:44.992848 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.038968 2557 apiserver.go:52] "Watching apiserver" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.060992 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.064240 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.064629 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-2cnzk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal","openshift-network-diagnostics/network-check-target-kz9ls","openshift-network-operator/iptables-alerter-9jt6j","openshift-ovn-kubernetes/ovnkube-node-x5ll6","kube-system/global-pull-secret-syncer-wz8mb","kube-system/konnectivity-agent-f6rxm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g","openshift-multus/multus-additional-cni-plugins-jf4qg","openshift-multus/multus-hnxq7","openshift-multus/network-metrics-daemon-q4sg5","kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rhx4w","openshift-dns/node-resolver-z6bxk"] Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.066272 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.067490 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.067558 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.068720 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.068776 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.069900 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.069954 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7l7f9\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.070025 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.070121 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.070419 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.072327 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.073906 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.075304 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.075365 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.075422 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.075445 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 02:17:45.107186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.075365 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wtl6t\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.076637 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7l6j8\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.076909 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.076965 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.077164 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.077306 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.078790 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.078790 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.078843 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.079990 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8mlzf\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.080166 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.080212 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.080264 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.081178 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.081371 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tqzxv\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.082312 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.084990 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.087548 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.088920 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.089162 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.089162 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.090088 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.091094 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 02:12:44 +0000 UTC" deadline="2028-02-01 23:11:13.96014523 +0000 UTC" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.091127 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13796h53m28.869022097s" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.094352 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.094417 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.094714 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.094719 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.094792 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.094928 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.095732 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.095777 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.095849 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-68wjn\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.095932 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.096012 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z5n9w\"" Jul 07 02:17:45.109088 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.096075 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-45rcd\"" Jul 07 02:17:45.111191 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.096141 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 02:17:45.111191 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.097324 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4m52t\"" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.135525 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5x74l" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.147175 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5x74l" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.163017 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168227 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspc6\" (UniqueName: \"kubernetes.io/projected/2de54061-29f9-4edb-8a38-28f97eb52745-kube-api-access-jspc6\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-cni-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168260 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxbl\" (UniqueName: \"kubernetes.io/projected/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-kube-api-access-qpxbl\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168280 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysctl-d\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168339 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-ovn\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovnkube-script-lib\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168398 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-cnibin\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168419 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c54b438a-acbb-4abe-81f5-c46b56541886-konnectivity-ca\") pod \"konnectivity-agent-f6rxm\" (UID: \"c54b438a-acbb-4abe-81f5-c46b56541886\") " pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168437 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-sys-fs\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168483 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-os-release\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168508 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mss\" (UniqueName: \"kubernetes.io/projected/2379c48f-d209-40e9-88da-f7427e4e9a18-kube-api-access-d6mss\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-iptables-alerter-script\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168544 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-run\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.302129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168561 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-var-lib-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-node-log\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-system-cni-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168614 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysconfig\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168628 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-sys\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168643 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-kubelet\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168675 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-daemon-config\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168690 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-systemd\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168705 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-etc-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168721 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovnkube-config\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c68a2cba-9963-440a-96f3-841863503bee-host\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168751 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06baddfd-0455-4b6d-8e02-c9fef640b572-kubelet-config\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168775 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-cni-binary-copy\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-conf-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168839 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-lib-modules\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168875 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.302843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168899 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-socket-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168915 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/c68a2cba-9963-440a-96f3-841863503bee-kube-api-access-pfd6r\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168932 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6pm\" (UniqueName: \"kubernetes.io/projected/fc14b32a-df85-420d-a697-67612b30ac22-kube-api-access-2r6pm\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168956 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-modprobe-d\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.168996 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysctl-conf\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169011 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-systemd\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169032 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-host\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mth\" (UniqueName: \"kubernetes.io/projected/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-kube-api-access-76mth\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169065 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-k8s-cni-cncf-io\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169087 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-multus-certs\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169121 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-etc-kubernetes\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169147 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-tuned\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169166 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jcss\" (UniqueName: \"kubernetes.io/projected/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-kube-api-access-6jcss\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169197 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-systemd-units\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169213 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-run-netns\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169228 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4vp\" (UniqueName: \"kubernetes.io/projected/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-kube-api-access-vg4vp\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.303516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169250 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169267 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-cni-multus\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169307 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169330 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-registration-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169345 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a949319-84f9-4682-9f53-012572cc79ea-hosts-file\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169360 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmzz\" (UniqueName: \"kubernetes.io/projected/6a949319-84f9-4682-9f53-012572cc79ea-kube-api-access-chmzz\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169374 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-cnibin\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169407 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-slash\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169423 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169438 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-log-socket\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169460 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169488 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-system-cni-dir\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169506 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-kubelet\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169522 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169551 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-tmp\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.304116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169576 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c68a2cba-9963-440a-96f3-841863503bee-serviceca\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06baddfd-0455-4b6d-8e02-c9fef640b572-dbus\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169614 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-hostroot\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169636 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-var-lib-kubelet\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169658 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-cni-bin\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169676 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovn-node-metrics-cert\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c54b438a-acbb-4abe-81f5-c46b56541886-agent-certs\") pod \"konnectivity-agent-f6rxm\" (UID: \"c54b438a-acbb-4abe-81f5-c46b56541886\") " pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169710 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-device-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169727 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-netns\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169743 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-cni-bin\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169760 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-os-release\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169786 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169816 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-kubernetes\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169858 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169876 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a949319-84f9-4682-9f53-012572cc79ea-tmp-dir\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.304745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169891 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-socket-dir-parent\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169907 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-host-slash\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169932 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-cni-netd\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.169953 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-env-overrides\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.269672 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0889d5315af6f41b8c9a541959d87a.slice/crio-419174d106255d40b3b3e1392aec73af3bc3b97587feff4092cd1c8a2a013bfb WatchSource:0}: Error finding container 419174d106255d40b3b3e1392aec73af3bc3b97587feff4092cd1c8a2a013bfb: Status 404 returned error can't find the container with id 419174d106255d40b3b3e1392aec73af3bc3b97587feff4092cd1c8a2a013bfb Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270165 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-kubelet\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270195 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.270219 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7fbc4b9b8b263ad709e9ac1b28ed97.slice/crio-71ab025230cc2f8c38af4d1de4ed9421df62b45d3c2d527e323ad338c689853a WatchSource:0}: Error finding container 71ab025230cc2f8c38af4d1de4ed9421df62b45d3c2d527e323ad338c689853a: Status 404 returned error can't find the container with id 71ab025230cc2f8c38af4d1de4ed9421df62b45d3c2d527e323ad338c689853a Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-daemon-config\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270262 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-systemd\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270271 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-kubelet\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270287 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-etc-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270317 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovnkube-config\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270332 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-etc-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.270350 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270367 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-systemd\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.305338 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.270459 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret podName:06baddfd-0455-4b6d-8e02-c9fef640b572 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:45.770401072 +0000 UTC m=+2.133603654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret") pod "global-pull-secret-syncer-wz8mb" (UID: "06baddfd-0455-4b6d-8e02-c9fef640b572") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270485 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c68a2cba-9963-440a-96f3-841863503bee-host\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270519 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06baddfd-0455-4b6d-8e02-c9fef640b572-kubelet-config\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270539 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c68a2cba-9963-440a-96f3-841863503bee-host\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270547 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-cni-binary-copy\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-conf-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270609 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-lib-modules\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270618 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06baddfd-0455-4b6d-8e02-c9fef640b572-kubelet-config\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270635 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270663 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-socket-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270669 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-conf-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270787 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-socket-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270793 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-lib-modules\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270818 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-daemon-config\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270842 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/c68a2cba-9963-440a-96f3-841863503bee-kube-api-access-pfd6r\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270879 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6pm\" (UniqueName: \"kubernetes.io/projected/fc14b32a-df85-420d-a697-67612b30ac22-kube-api-access-2r6pm\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-modprobe-d\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.305969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysctl-conf\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270941 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-systemd\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270946 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovnkube-config\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.270964 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-host\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271014 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-host\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271017 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-modprobe-d\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271030 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-cni-binary-copy\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271063 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-systemd\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271114 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysctl-conf\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mth\" (UniqueName: \"kubernetes.io/projected/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-kube-api-access-76mth\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271187 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-k8s-cni-cncf-io\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271192 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271216 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-multus-certs\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271239 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-k8s-cni-cncf-io\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271242 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-etc-kubernetes\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271269 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-tuned\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271280 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-multus-certs\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271288 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-etc-kubernetes\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.306579 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271295 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jcss\" (UniqueName: \"kubernetes.io/projected/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-kube-api-access-6jcss\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271321 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-systemd-units\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-run-netns\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271368 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4vp\" (UniqueName: \"kubernetes.io/projected/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-kube-api-access-vg4vp\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271387 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271406 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-systemd-units\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271419 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-cni-multus\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271426 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-run-netns\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271440 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271481 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-cni-multus\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271493 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271523 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271560 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-registration-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271558 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271614 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-registration-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a949319-84f9-4682-9f53-012572cc79ea-hosts-file\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271684 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmzz\" (UniqueName: \"kubernetes.io/projected/6a949319-84f9-4682-9f53-012572cc79ea-kube-api-access-chmzz\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.307335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271697 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a949319-84f9-4682-9f53-012572cc79ea-hosts-file\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271711 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-cnibin\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271742 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-slash\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271767 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-cnibin\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271805 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271822 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-log-socket\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271843 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-slash\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271883 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-system-cni-dir\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271901 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-log-socket\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271930 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271937 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-kubelet\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271941 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271966 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-system-cni-dir\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271970 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271992 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-kubelet\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.307902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272006 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc14b32a-df85-420d-a697-67612b30ac22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.271999 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-tmp\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c68a2cba-9963-440a-96f3-841863503bee-serviceca\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.272070 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06baddfd-0455-4b6d-8e02-c9fef640b572-dbus\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272109 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-hostroot\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.272128 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs podName:2379c48f-d209-40e9-88da-f7427e4e9a18 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:45.772109014 +0000 UTC m=+2.135311591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs") pod "network-metrics-daemon-q4sg5" (UID: "2379c48f-d209-40e9-88da-f7427e4e9a18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272150 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-var-lib-kubelet\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272168 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-cni-bin\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272189 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovn-node-metrics-cert\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c54b438a-acbb-4abe-81f5-c46b56541886-agent-certs\") pod \"konnectivity-agent-f6rxm\" (UID: \"c54b438a-acbb-4abe-81f5-c46b56541886\") " pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272234 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-var-lib-kubelet\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-device-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272281 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06baddfd-0455-4b6d-8e02-c9fef640b572-dbus\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272282 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-netns\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272320 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-run-netns\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-cni-bin\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.308536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272361 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-os-release\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272389 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-kubernetes\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272461 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272512 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a949319-84f9-4682-9f53-012572cc79ea-tmp-dir\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272541 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-socket-dir-parent\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272557 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c68a2cba-9963-440a-96f3-841863503bee-serviceca\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272576 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-host-slash\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272603 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-cni-netd\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272619 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-hostroot\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-env-overrides\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272659 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272665 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-device-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272672 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272689 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspc6\" (UniqueName: \"kubernetes.io/projected/2de54061-29f9-4edb-8a38-28f97eb52745-kube-api-access-jspc6\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272801 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-os-release\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.309073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272854 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-cni-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272867 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-kubernetes\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272883 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxbl\" (UniqueName: \"kubernetes.io/projected/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-kube-api-access-qpxbl\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272907 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-host-slash\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272910 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysctl-d\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272936 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-ovn\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272963 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovnkube-script-lib\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.272939 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-cni-bin\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273034 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-cnibin\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273061 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c54b438a-acbb-4abe-81f5-c46b56541886-konnectivity-ca\") pod \"konnectivity-agent-f6rxm\" (UID: \"c54b438a-acbb-4abe-81f5-c46b56541886\") " pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273075 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a949319-84f9-4682-9f53-012572cc79ea-tmp-dir\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273100 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-sys-fs\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273128 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-os-release\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273148 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-socket-dir-parent\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mss\" (UniqueName: \"kubernetes.io/projected/2379c48f-d209-40e9-88da-f7427e4e9a18-kube-api-access-d6mss\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273192 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-host-var-lib-cni-bin\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273193 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-iptables-alerter-script\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.309708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273223 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273232 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-run\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273288 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-env-overrides\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-var-lib-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273330 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-host-cni-netd\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273337 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-node-log\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273384 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-node-log\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273420 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-run-ovn\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273519 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-sys-fs\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273554 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-system-cni-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273586 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysconfig\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273652 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-run\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273704 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-multus-cni-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273732 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-sys\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273759 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-os-release\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273794 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc14b32a-df85-420d-a697-67612b30ac22-cnibin\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273816 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-sys\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273815 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-iptables-alerter-script\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.310248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273847 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysconfig\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273923 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-sysctl-d\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.273984 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-system-cni-dir\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.274058 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2de54061-29f9-4edb-8a38-28f97eb52745-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.274095 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-var-lib-openvswitch\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.274290 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c54b438a-acbb-4abe-81f5-c46b56541886-konnectivity-ca\") pod \"konnectivity-agent-f6rxm\" (UID: \"c54b438a-acbb-4abe-81f5-c46b56541886\") " pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.274912 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovnkube-script-lib\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.275554 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-tmp\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.275691 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-etc-tuned\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.275714 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-ovn-node-metrics-cert\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.275879 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c54b438a-acbb-4abe-81f5-c46b56541886-agent-certs\") pod \"konnectivity-agent-f6rxm\" (UID: \"c54b438a-acbb-4abe-81f5-c46b56541886\") " pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.276474 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.280135 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.280157 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.280171 2557 projected.go:196] Error preparing data for projected volume kube-api-access-hx2b9 for pod openshift-network-diagnostics/network-check-target-kz9ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.280236 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9 podName:3d82edaa-902a-4dd4-a5f3-acb327cfb733 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:45.780214469 +0000 UTC m=+2.143417037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hx2b9" (UniqueName: "kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9") pod "network-check-target-kz9ls" (UID: "3d82edaa-902a-4dd4-a5f3-acb327cfb733") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.280356 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/c68a2cba-9963-440a-96f3-841863503bee-kube-api-access-pfd6r\") pod \"node-ca-2cnzk\" (UID: \"c68a2cba-9963-440a-96f3-841863503bee\") " pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.281100 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6pm\" (UniqueName: \"kubernetes.io/projected/fc14b32a-df85-420d-a697-67612b30ac22-kube-api-access-2r6pm\") pod \"multus-additional-cni-plugins-jf4qg\" (UID: \"fc14b32a-df85-420d-a697-67612b30ac22\") " pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.310962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.281166 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mth\" (UniqueName: \"kubernetes.io/projected/a13a3ca4-ecdd-40a3-a2ea-af7971aeae21-kube-api-access-76mth\") pod \"iptables-alerter-9jt6j\" (UID: \"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21\") " pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.311918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.282485 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmzz\" (UniqueName: \"kubernetes.io/projected/6a949319-84f9-4682-9f53-012572cc79ea-kube-api-access-chmzz\") pod \"node-resolver-z6bxk\" (UID: \"6a949319-84f9-4682-9f53-012572cc79ea\") " pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.311918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.283285 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mss\" (UniqueName: \"kubernetes.io/projected/2379c48f-d209-40e9-88da-f7427e4e9a18-kube-api-access-d6mss\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:45.311918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.283409 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspc6\" (UniqueName: \"kubernetes.io/projected/2de54061-29f9-4edb-8a38-28f97eb52745-kube-api-access-jspc6\") pod \"aws-ebs-csi-driver-node-qzs7g\" (UID: \"2de54061-29f9-4edb-8a38-28f97eb52745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.311918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.284209 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4vp\" (UniqueName: \"kubernetes.io/projected/888ed7b5-198b-4f2a-812c-fc6e8122d6e6-kube-api-access-vg4vp\") pod \"ovnkube-node-x5ll6\" (UID: \"888ed7b5-198b-4f2a-812c-fc6e8122d6e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.311918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.284692 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxbl\" (UniqueName: \"kubernetes.io/projected/a5bd485a-dadb-4dfc-90b5-a1134ffe688a-kube-api-access-qpxbl\") pod \"multus-hnxq7\" (UID: \"a5bd485a-dadb-4dfc-90b5-a1134ffe688a\") " pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.311918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.286387 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jcss\" (UniqueName: \"kubernetes.io/projected/8f997eee-ce3e-44d6-9d5a-b42eae332ea6-kube-api-access-6jcss\") pod \"tuned-rhx4w\" (UID: \"8f997eee-ce3e-44d6-9d5a-b42eae332ea6\") " pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.314590 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.397761 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hnxq7" Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.409119 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bd485a_dadb_4dfc_90b5_a1134ffe688a.slice/crio-a0cbba8834ce56311d716237a3676a92d301783469424ecc372f34abb32a605d WatchSource:0}: Error finding container a0cbba8834ce56311d716237a3676a92d301783469424ecc372f34abb32a605d: Status 404 returned error can't find the container with id a0cbba8834ce56311d716237a3676a92d301783469424ecc372f34abb32a605d Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.419096 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9jt6j" Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.427802 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13a3ca4_ecdd_40a3_a2ea_af7971aeae21.slice/crio-a7cf599c95fac2db83d30f40745de4c8da654d9ea63394252e512c8f9e28ad0a WatchSource:0}: Error finding container a7cf599c95fac2db83d30f40745de4c8da654d9ea63394252e512c8f9e28ad0a: Status 404 returned error can't find the container with id a7cf599c95fac2db83d30f40745de4c8da654d9ea63394252e512c8f9e28ad0a Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.459114 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cnzk" Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.462340 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.468409 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68a2cba_9963_440a_96f3_841863503bee.slice/crio-b8e7366f34b159c61df9f5aa472057f40c51ec08c8a383d5d04da024e4bde721 WatchSource:0}: Error finding container b8e7366f34b159c61df9f5aa472057f40c51ec08c8a383d5d04da024e4bde721: Status 404 returned error can't find the container with id b8e7366f34b159c61df9f5aa472057f40c51ec08c8a383d5d04da024e4bde721 Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.472423 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f997eee_ce3e_44d6_9d5a_b42eae332ea6.slice/crio-e184088563e4cd0cc75817b80fc0d9d53a3d41d76798235852b1ff978c8a0b20 WatchSource:0}: Error finding container e184088563e4cd0cc75817b80fc0d9d53a3d41d76798235852b1ff978c8a0b20: Status 404 returned error can't find the container with id e184088563e4cd0cc75817b80fc0d9d53a3d41d76798235852b1ff978c8a0b20 Jul 07 02:17:45.509848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.500785 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.514904 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc14b32a_df85_420d_a697_67612b30ac22.slice/crio-81b5262941afd0552dd20133ec1223e2d0a5a8972d5146b038edc731a48bfdb4 WatchSource:0}: Error finding container 81b5262941afd0552dd20133ec1223e2d0a5a8972d5146b038edc731a48bfdb4: Status 404 returned error can't find the container with id 81b5262941afd0552dd20133ec1223e2d0a5a8972d5146b038edc731a48bfdb4 Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.519124 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.528086 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888ed7b5_198b_4f2a_812c_fc6e8122d6e6.slice/crio-0954cdb79d665dd63dd41ff834089d8e877f055fa48cbbcb5c5ab4b17be43803 WatchSource:0}: Error finding container 0954cdb79d665dd63dd41ff834089d8e877f055fa48cbbcb5c5ab4b17be43803: Status 404 returned error can't find the container with id 0954cdb79d665dd63dd41ff834089d8e877f055fa48cbbcb5c5ab4b17be43803 Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.539415 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.546941 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54b438a_acbb_4abe_81f5_c46b56541886.slice/crio-995790852ebefb283fdeb5b577c75c794d5436e5d7e54f0177c4d25f7cd8d806 WatchSource:0}: Error finding container 995790852ebefb283fdeb5b577c75c794d5436e5d7e54f0177c4d25f7cd8d806: Status 404 returned error can't find the container with id 995790852ebefb283fdeb5b577c75c794d5436e5d7e54f0177c4d25f7cd8d806 Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.557723 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.567775 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de54061_29f9_4edb_8a38_28f97eb52745.slice/crio-e3a95139af926d474acbdee55190743c828370684d7299204063f6edea8cbe45 WatchSource:0}: Error finding container e3a95139af926d474acbdee55190743c828370684d7299204063f6edea8cbe45: Status 404 returned error can't find the container with id e3a95139af926d474acbdee55190743c828370684d7299204063f6edea8cbe45 Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.568731 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z6bxk" Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:17:45.576863 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a949319_84f9_4682_9f53_012572cc79ea.slice/crio-fb91a247a00c6e3f279184a96a20a371d68673c477bb4fd149f29e148e8f3279 WatchSource:0}: Error finding container fb91a247a00c6e3f279184a96a20a371d68673c477bb4fd149f29e148e8f3279: Status 404 returned error can't find the container with id fb91a247a00c6e3f279184a96a20a371d68673c477bb4fd149f29e148e8f3279 Jul 07 02:17:45.777112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.602915 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:45.778859 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.778531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:45.778859 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.778722 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:45.778859 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.778797 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret podName:06baddfd-0455-4b6d-8e02-c9fef640b572 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:46.778775849 +0000 UTC m=+3.141978412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret") pod "global-pull-secret-syncer-wz8mb" (UID: "06baddfd-0455-4b6d-8e02-c9fef640b572") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:45.778859 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.778672 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:45.779411 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.779052 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:45.779411 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.779107 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs podName:2379c48f-d209-40e9-88da-f7427e4e9a18 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:46.779089612 +0000 UTC m=+3.142292175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs") pod "network-metrics-daemon-q4sg5" (UID: "2379c48f-d209-40e9-88da-f7427e4e9a18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:45.919796 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:45.880059 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:45.919796 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.880291 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:45.919796 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.880321 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:45.919796 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.880336 2557 projected.go:196] Error preparing data for projected volume kube-api-access-hx2b9 for pod openshift-network-diagnostics/network-check-target-kz9ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:45.919796 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:45.880423 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9 podName:3d82edaa-902a-4dd4-a5f3-acb327cfb733 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:46.880387662 +0000 UTC m=+3.243590244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hx2b9" (UniqueName: "kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9") pod "network-check-target-kz9ls" (UID: "3d82edaa-902a-4dd4-a5f3-acb327cfb733") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:46.164102 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.149879 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:45 +0000 UTC" deadline="2028-02-14 08:50:33.511404421 +0000 UTC" Jul 07 02:17:46.164102 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.149914 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14094h32m47.36149488s" Jul 07 02:17:46.202908 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.196854 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:46.202908 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.196990 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:46.202908 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.200035 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:46.266209 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.221970 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cnzk" event={"ID":"c68a2cba-9963-440a-96f3-841863503bee","Type":"ContainerStarted","Data":"b8e7366f34b159c61df9f5aa472057f40c51ec08c8a383d5d04da024e4bde721"} Jul 07 02:17:46.266209 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.246771 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9jt6j" event={"ID":"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21","Type":"ContainerStarted","Data":"a7cf599c95fac2db83d30f40745de4c8da654d9ea63394252e512c8f9e28ad0a"} Jul 07 02:17:46.266209 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.257850 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" event={"ID":"fe7fbc4b9b8b263ad709e9ac1b28ed97","Type":"ContainerStarted","Data":"71ab025230cc2f8c38af4d1de4ed9421df62b45d3c2d527e323ad338c689853a"} Jul 07 02:17:46.266209 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.266142 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" event={"ID":"2de54061-29f9-4edb-8a38-28f97eb52745","Type":"ContainerStarted","Data":"e3a95139af926d474acbdee55190743c828370684d7299204063f6edea8cbe45"} Jul 07 02:17:46.268242 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.267885 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6rxm" event={"ID":"c54b438a-acbb-4abe-81f5-c46b56541886","Type":"ContainerStarted","Data":"995790852ebefb283fdeb5b577c75c794d5436e5d7e54f0177c4d25f7cd8d806"} Jul 07 02:17:46.339654 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.273120 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"0954cdb79d665dd63dd41ff834089d8e877f055fa48cbbcb5c5ab4b17be43803"} Jul 07 02:17:46.339654 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.282718 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hnxq7" event={"ID":"a5bd485a-dadb-4dfc-90b5-a1134ffe688a","Type":"ContainerStarted","Data":"a0cbba8834ce56311d716237a3676a92d301783469424ecc372f34abb32a605d"} Jul 07 02:17:46.339654 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.285942 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"bc0889d5315af6f41b8c9a541959d87a","Type":"ContainerStarted","Data":"419174d106255d40b3b3e1392aec73af3bc3b97587feff4092cd1c8a2a013bfb"} Jul 07 02:17:46.339654 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.294857 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z6bxk" event={"ID":"6a949319-84f9-4682-9f53-012572cc79ea","Type":"ContainerStarted","Data":"fb91a247a00c6e3f279184a96a20a371d68673c477bb4fd149f29e148e8f3279"} Jul 07 02:17:46.339654 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.312547 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerStarted","Data":"81b5262941afd0552dd20133ec1223e2d0a5a8972d5146b038edc731a48bfdb4"} Jul 07 02:17:46.339654 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.317625 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" event={"ID":"8f997eee-ce3e-44d6-9d5a-b42eae332ea6","Type":"ContainerStarted","Data":"e184088563e4cd0cc75817b80fc0d9d53a3d41d76798235852b1ff978c8a0b20"} Jul 07 02:17:46.789898 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.789014 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:46.789898 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.789107 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:46.789898 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.789253 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:46.789898 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.789342 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret podName:06baddfd-0455-4b6d-8e02-c9fef640b572 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.789324018 +0000 UTC m=+5.152526585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret") pod "global-pull-secret-syncer-wz8mb" (UID: "06baddfd-0455-4b6d-8e02-c9fef640b572") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:46.789898 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.789798 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:46.789898 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.789854 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs podName:2379c48f-d209-40e9-88da-f7427e4e9a18 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.789835921 +0000 UTC m=+5.153038488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs") pod "network-metrics-daemon-q4sg5" (UID: "2379c48f-d209-40e9-88da-f7427e4e9a18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:46.889882 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:46.889837 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:46.890088 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.890064 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:46.890088 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.890088 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:46.890209 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.890102 2557 projected.go:196] Error preparing data for projected volume kube-api-access-hx2b9 for pod openshift-network-diagnostics/network-check-target-kz9ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:46.890558 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:46.890534 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9 podName:3d82edaa-902a-4dd4-a5f3-acb327cfb733 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.890145727 +0000 UTC m=+5.253348306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hx2b9" (UniqueName: "kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9") pod "network-check-target-kz9ls" (UID: "3d82edaa-902a-4dd4-a5f3-acb327cfb733") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:47.150489 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:47.150303 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:45 +0000 UTC" deadline="2028-01-18 08:43:01.642421606 +0000 UTC" Jul 07 02:17:47.150489 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:47.150391 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13446h25m14.492050797s" Jul 07 02:17:47.195125 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:47.194406 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:47.195125 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:47.194550 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:17:47.195125 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:47.194963 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:47.195125 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:47.195058 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:47.566571 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:47.566373 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:48.195672 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:48.195104 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:48.195672 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.195272 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:48.809436 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:48.809338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:48.809637 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:48.809431 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:48.809637 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.809605 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:48.809748 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.809672 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs podName:2379c48f-d209-40e9-88da-f7427e4e9a18 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:52.80965184 +0000 UTC m=+9.172854424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs") pod "network-metrics-daemon-q4sg5" (UID: "2379c48f-d209-40e9-88da-f7427e4e9a18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:48.810098 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.809997 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:48.810098 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.810073 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret podName:06baddfd-0455-4b6d-8e02-c9fef640b572 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:52.810054318 +0000 UTC m=+9.173256896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret") pod "global-pull-secret-syncer-wz8mb" (UID: "06baddfd-0455-4b6d-8e02-c9fef640b572") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:48.910149 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:48.910109 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:48.910372 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.910297 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:48.910436 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.910383 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:48.910436 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.910400 2557 projected.go:196] Error preparing data for projected volume kube-api-access-hx2b9 for pod openshift-network-diagnostics/network-check-target-kz9ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:48.910568 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:48.910487 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9 podName:3d82edaa-902a-4dd4-a5f3-acb327cfb733 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:52.91046408 +0000 UTC m=+9.273666659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hx2b9" (UniqueName: "kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9") pod "network-check-target-kz9ls" (UID: "3d82edaa-902a-4dd4-a5f3-acb327cfb733") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:49.195282 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:49.195191 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:49.195487 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:49.195351 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:17:49.195806 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:49.195788 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:49.196183 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:49.195890 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:50.195805 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:50.195215 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:50.195805 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:50.195377 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:51.194651 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:51.194614 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:51.194857 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:51.194746 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:51.194857 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:51.194614 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:51.194977 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:51.194918 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:17:52.194949 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:52.194916 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:52.195400 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.195062 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:52.848311 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:52.847974 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:52.848311 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:52.848047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:52.848311 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.848139 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:52.848311 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.848169 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:52.848311 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.848210 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs podName:2379c48f-d209-40e9-88da-f7427e4e9a18 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:00.848191788 +0000 UTC m=+17.211394367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs") pod "network-metrics-daemon-q4sg5" (UID: "2379c48f-d209-40e9-88da-f7427e4e9a18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:52.848311 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.848229 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret podName:06baddfd-0455-4b6d-8e02-c9fef640b572 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:00.84821945 +0000 UTC m=+17.211422019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret") pod "global-pull-secret-syncer-wz8mb" (UID: "06baddfd-0455-4b6d-8e02-c9fef640b572") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:52.948926 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:52.948876 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:52.949090 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.949073 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:52.949165 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.949100 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:52.949165 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.949113 2557 projected.go:196] Error preparing data for projected volume kube-api-access-hx2b9 for pod openshift-network-diagnostics/network-check-target-kz9ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:52.949257 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:52.949176 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9 podName:3d82edaa-902a-4dd4-a5f3-acb327cfb733 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:00.949156014 +0000 UTC m=+17.312358591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hx2b9" (UniqueName: "kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9") pod "network-check-target-kz9ls" (UID: "3d82edaa-902a-4dd4-a5f3-acb327cfb733") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:53.194698 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:53.194158 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:53.194698 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:53.194296 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:53.194698 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:53.194352 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:53.194698 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:53.194476 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:17:54.198188 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:54.197922 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:54.198188 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:54.198057 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:55.194994 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:55.194959 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:55.195201 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:55.195095 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:17:55.195415 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:55.195388 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:55.195551 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:55.195505 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:56.198307 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:56.198277 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:56.198721 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:56.198429 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:57.194411 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:57.194200 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:57.194577 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:57.194218 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:57.194577 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:57.194504 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:17:57.194577 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:57.194542 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:58.194559 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:58.194475 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:17:58.194985 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:58.194618 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:17:59.194137 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:59.194111 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:17:59.194346 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:17:59.194110 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:17:59.194346 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:59.194225 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:17:59.194346 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:17:59.194305 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:00.195101 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:00.195071 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:00.195546 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:00.195206 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:00.908286 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:00.908246 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:00.908536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:00.908315 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:00.908536 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:00.908435 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:00.908536 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:00.908508 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs podName:2379c48f-d209-40e9-88da-f7427e4e9a18 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:16.908490812 +0000 UTC m=+33.271693395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs") pod "network-metrics-daemon-q4sg5" (UID: "2379c48f-d209-40e9-88da-f7427e4e9a18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:00.908536 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:00.908434 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:00.908752 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:00.908595 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret podName:06baddfd-0455-4b6d-8e02-c9fef640b572 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:16.90857404 +0000 UTC m=+33.271776612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret") pod "global-pull-secret-syncer-wz8mb" (UID: "06baddfd-0455-4b6d-8e02-c9fef640b572") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:01.009282 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:01.009245 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:01.009463 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:01.009412 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:18:01.009463 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:01.009429 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:18:01.009463 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:01.009440 2557 projected.go:196] Error preparing data for projected volume kube-api-access-hx2b9 for pod openshift-network-diagnostics/network-check-target-kz9ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:01.009598 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:01.009490 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9 podName:3d82edaa-902a-4dd4-a5f3-acb327cfb733 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:17.009477702 +0000 UTC m=+33.372680264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hx2b9" (UniqueName: "kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9") pod "network-check-target-kz9ls" (UID: "3d82edaa-902a-4dd4-a5f3-acb327cfb733") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:01.195205 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:01.195125 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:01.195205 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:01.195164 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:01.195654 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:01.195263 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:01.195654 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:01.195395 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:02.195337 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:02.195301 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:02.195766 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:02.195469 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:03.194589 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:03.194549 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:03.194765 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:03.194604 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:03.194765 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:03.194668 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:03.194887 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:03.194807 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:04.194838 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:04.194794 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:04.195287 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:04.194930 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:05.195084 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:05.195039 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:05.195587 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:05.195051 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:05.195587 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:05.195157 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:05.195587 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:05.195250 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:06.194344 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:06.194320 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:06.194475 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:06.194455 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:06.364504 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:06.364469 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hnxq7" event={"ID":"a5bd485a-dadb-4dfc-90b5-a1134ffe688a","Type":"ContainerStarted","Data":"bc1ebea4fce5d99ef3226aa50060af4d5fe1c9bedf6d00046476e6914cefdde9"} Jul 07 02:18:06.366157 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:06.366131 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" event={"ID":"8f997eee-ce3e-44d6-9d5a-b42eae332ea6","Type":"ContainerStarted","Data":"569c860458f8c8ea0e78c4fe7e4469cbc7471dd01865af1e7653d25312cc1737"} Jul 07 02:18:06.367725 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:06.367700 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" event={"ID":"fe7fbc4b9b8b263ad709e9ac1b28ed97","Type":"ContainerStarted","Data":"d4f4263cd6ac76280703d9e36f24c6d340760e63fee2f9d61256791d37624229"} Jul 07 02:18:06.383915 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:06.383876 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hnxq7" podStartSLOduration=1.740816908 podStartE2EDuration="22.383863659s" totalImagesPullingTime="20.643046751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:06.383575928 +0000 UTC m=+22.746778511" watchObservedRunningTime="2026-07-07 02:18:06.383863659 +0000 UTC m=+22.747066243" Jul 07 02:18:06.399753 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:06.398047 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-74.ec2.internal" podStartSLOduration=21.398027513 podStartE2EDuration="21.398027513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:06.397044354 +0000 UTC m=+22.760246939" watchObservedRunningTime="2026-07-07 02:18:06.398027513 +0000 UTC m=+22.761230098" Jul 07 02:18:06.420900 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:06.420839 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rhx4w" podStartSLOduration=1.858818343 podStartE2EDuration="22.420816068s" totalImagesPullingTime="20.561997725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:06.419924141 +0000 UTC m=+22.783126736" watchObservedRunningTime="2026-07-07 02:18:06.420816068 +0000 UTC m=+22.784018653" Jul 07 02:18:07.195318 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.195145 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:07.195464 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.195146 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:07.195464 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:07.195434 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:07.195588 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:07.195493 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:07.371132 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.370921 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6rxm" event={"ID":"c54b438a-acbb-4abe-81f5-c46b56541886","Type":"ContainerStarted","Data":"806d42413103913a71ea7ed0f87af8c0e0f615481bd59c6eeffb6489b8ed095f"} Jul 07 02:18:07.374147 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.374112 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"4ddb07efe1791b94721b989906c0e676e035851e17c38c3773219d726c2766b8"} Jul 07 02:18:07.374297 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.374190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"1e18fc23225e56c42abe7f3320013ee165849dd11a67a289c1e1cbaa34bd9a7a"} Jul 07 02:18:07.374297 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.374230 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"66b2b9ad74dd1afb444efd7044cf156eae75368207c5e29256712bc7d67b6131"} Jul 07 02:18:07.374297 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.374266 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"e278fc68e3d36eae2d18f1cc8d8242d26c891e00ccbb63162d5945c7c4027a31"} Jul 07 02:18:07.374460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.374310 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"a45de7f194fc7fb99c410509ae3a8f12c60d5fe9b1048d3d81c1a42efe38bfb8"} Jul 07 02:18:07.374460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.374353 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"b7639481d9f00ec8ed9cfd6f50fd5879c378b192db2b46c1eaa4ed8067042997"} Jul 07 02:18:07.375574 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.375550 2557 generic.go:356] "Generic (PLEG): container finished" podID="bc0889d5315af6f41b8c9a541959d87a" containerID="53edb4cdfc7a1d7fdc530a6fffa3c512ca5dc572bf6f98273f741d09b830ffe0" exitCode=0 Jul 07 02:18:07.375678 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.375618 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"bc0889d5315af6f41b8c9a541959d87a","Type":"ContainerDied","Data":"53edb4cdfc7a1d7fdc530a6fffa3c512ca5dc572bf6f98273f741d09b830ffe0"} Jul 07 02:18:07.377168 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.377143 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z6bxk" event={"ID":"6a949319-84f9-4682-9f53-012572cc79ea","Type":"ContainerStarted","Data":"7651c8b6f15cd278c0e661f4a99b412ffa31ec84bd65e2c73f1e544dc7915ad0"} Jul 07 02:18:07.378639 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.378617 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc14b32a-df85-420d-a697-67612b30ac22" containerID="b043079d8366d6848214cc149b2395d75db57f750c3a001c40eadc645edae0be" exitCode=0 Jul 07 02:18:07.378766 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.378682 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerDied","Data":"b043079d8366d6848214cc149b2395d75db57f750c3a001c40eadc645edae0be"} Jul 07 02:18:07.380192 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.380169 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cnzk" event={"ID":"c68a2cba-9963-440a-96f3-841863503bee","Type":"ContainerStarted","Data":"3af119a1634170a1027ce6845e5da4d2d508c8a671d940e4deb8ebb9c260e1b0"} Jul 07 02:18:07.381647 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.381623 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" event={"ID":"2de54061-29f9-4edb-8a38-28f97eb52745","Type":"ContainerStarted","Data":"d012172d20f8e51b44ab589aff3c380518d86d45cd1536826d776994f30c7871"} Jul 07 02:18:07.387295 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.387260 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f6rxm" podStartSLOduration=2.167546267 podStartE2EDuration="22.387248612s" totalImagesPullingTime="20.219702345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.386770858 +0000 UTC m=+23.749973447" watchObservedRunningTime="2026-07-07 02:18:07.387248612 +0000 UTC m=+23.750451195" Jul 07 02:18:07.404123 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.404084 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z6bxk" podStartSLOduration=1.975749073 podStartE2EDuration="22.404068112s" totalImagesPullingTime="20.428319039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.403818675 +0000 UTC m=+23.767021258" watchObservedRunningTime="2026-07-07 02:18:07.404068112 +0000 UTC m=+23.767270695" Jul 07 02:18:07.466647 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:07.466586 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2cnzk" podStartSLOduration=7.355745532 podStartE2EDuration="23.4665652s" totalImagesPullingTime="16.110819668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.466429658 +0000 UTC m=+23.829632242" watchObservedRunningTime="2026-07-07 02:18:07.4665652 +0000 UTC m=+23.829767785" Jul 07 02:18:08.194836 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:08.194804 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:08.195036 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:08.194947 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:08.386308 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:08.386264 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9jt6j" event={"ID":"a13a3ca4-ecdd-40a3-a2ea-af7971aeae21","Type":"ContainerStarted","Data":"fd95baf37d5495073127d2267011cec45d78ff9787e4c6c094bc2e97ebd226e9"} Jul 07 02:18:08.392653 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:08.392614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" event={"ID":"bc0889d5315af6f41b8c9a541959d87a","Type":"ContainerStarted","Data":"8c73576ad2f34715895ccec2ecb6a3e5f7a9c0435ae6728d8c2a5d7f3e14307e"} Jul 07 02:18:08.395848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:08.395824 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 02:18:08.403745 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:08.403702 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9jt6j" podStartSLOduration=4.065253874 podStartE2EDuration="24.403688444s" totalImagesPullingTime="20.33843457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.403417672 +0000 UTC m=+24.766620256" watchObservedRunningTime="2026-07-07 02:18:08.403688444 +0000 UTC m=+24.766891028" Jul 07 02:18:08.418704 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:08.418659 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-74.ec2.internal" podStartSLOduration=23.418639644 podStartE2EDuration="23.418639644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.418447868 +0000 UTC m=+24.781650454" watchObservedRunningTime="2026-07-07 02:18:08.418639644 +0000 UTC m=+24.781842231" Jul 07 02:18:09.148901 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.148784 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T02:18:08.395844061Z","UUID":"dfb98ec4-3ea3-4d99-b25c-81cac463c69c","Handler":null,"Name":"","Endpoint":""} Jul 07 02:18:09.150672 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.150630 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 02:18:09.150672 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.150673 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 02:18:09.194961 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.194928 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:09.195126 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:09.195031 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:09.195126 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.195096 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:09.195231 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:09.195209 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:09.397775 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.397667 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" event={"ID":"2de54061-29f9-4edb-8a38-28f97eb52745","Type":"ContainerStarted","Data":"dd46c52571c81d8462363ef65524dca877d5b2da53e8731beb5c784b5e1b86b1"} Jul 07 02:18:09.401839 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.401811 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"a5b6b89526717c66cf15eeb2d11eb1c7304835f654ac57fd5d9e69429c35070c"} Jul 07 02:18:09.650149 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.650065 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:18:09.650830 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:09.650804 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:18:10.194833 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:10.194646 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:10.195026 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:10.194957 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:10.405918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:10.405866 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" event={"ID":"2de54061-29f9-4edb-8a38-28f97eb52745","Type":"ContainerStarted","Data":"5aafde032a096414eaa753ff5724acec7b930171b05542c68b36b2a850a55bea"} Jul 07 02:18:10.433468 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:10.430446 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzs7g" podStartSLOduration=2.820356061 podStartE2EDuration="25.43042413s" totalImagesPullingTime="22.610068069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:10.429624991 +0000 UTC m=+26.792827573" watchObservedRunningTime="2026-07-07 02:18:10.43042413 +0000 UTC m=+26.793626716" Jul 07 02:18:11.194944 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:11.194913 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:11.195095 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:11.194925 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:11.195095 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:11.195045 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:11.195215 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:11.195128 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:11.412849 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:11.412767 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" event={"ID":"888ed7b5-198b-4f2a-812c-fc6e8122d6e6","Type":"ContainerStarted","Data":"bd7cd99e91a2d442dbef77a5fa4321b809943a790b9567584ad103d55b0182d2"} Jul 07 02:18:11.445483 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:11.445421 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" podStartSLOduration=5.642037073 podStartE2EDuration="26.445399319s" totalImagesPullingTime="20.803362246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:11.445292715 +0000 UTC m=+27.808495300" watchObservedRunningTime="2026-07-07 02:18:11.445399319 +0000 UTC m=+27.808601894" Jul 07 02:18:12.194848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:12.194817 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:12.195044 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:12.194939 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:12.415324 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:12.415284 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:18:12.415755 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:12.415397 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:18:12.415755 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:12.415446 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:18:12.431412 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:12.431370 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:18:12.431969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:12.431947 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:18:13.195490 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.195224 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:13.195665 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.195224 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:13.195665 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:13.195609 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:13.195797 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:13.195688 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:13.232898 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.232865 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wz8mb"] Jul 07 02:18:13.236251 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.236214 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kz9ls"] Jul 07 02:18:13.236904 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.236879 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q4sg5"] Jul 07 02:18:13.237019 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.236999 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:13.237132 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:13.237111 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:13.417424 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.417391 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:13.417424 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:13.417411 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:13.417878 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:13.417533 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:13.418045 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:13.418019 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:14.421116 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:14.421033 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc14b32a-df85-420d-a697-67612b30ac22" containerID="16352da480369270695cb8ac9cb09a1719c44ac338346a5add3d7c8a7964ed69" exitCode=0 Jul 07 02:18:14.421527 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:14.421112 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerDied","Data":"16352da480369270695cb8ac9cb09a1719c44ac338346a5add3d7c8a7964ed69"} Jul 07 02:18:14.942174 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:14.942134 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:18:14.942365 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:14.942329 2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 02:18:14.943195 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:14.943166 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f6rxm" Jul 07 02:18:15.194261 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:15.194165 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:15.194261 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:15.194194 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:15.194261 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:15.194194 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:15.194547 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:15.194273 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:15.194547 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:15.194359 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:15.194547 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:15.194447 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:16.427403 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:16.427358 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc14b32a-df85-420d-a697-67612b30ac22" containerID="11d9f2af70143003115aa9c183a42e05747430a720264a8c0b2a1dddf01903b8" exitCode=0 Jul 07 02:18:16.427842 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:16.427408 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerDied","Data":"11d9f2af70143003115aa9c183a42e05747430a720264a8c0b2a1dddf01903b8"} Jul 07 02:18:16.926670 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:16.926619 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:16.926850 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:16.926714 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:16.926850 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:16.926803 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:16.926850 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:16.926827 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:16.927077 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:16.926876 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs podName:2379c48f-d209-40e9-88da-f7427e4e9a18 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:48.926860442 +0000 UTC m=+65.290063004 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs") pod "network-metrics-daemon-q4sg5" (UID: "2379c48f-d209-40e9-88da-f7427e4e9a18") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:16.927077 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:16.926904 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret podName:06baddfd-0455-4b6d-8e02-c9fef640b572 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:48.926886109 +0000 UTC m=+65.290088671 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret") pod "global-pull-secret-syncer-wz8mb" (UID: "06baddfd-0455-4b6d-8e02-c9fef640b572") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:17.027458 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:17.027432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:17.027590 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:17.027576 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:18:17.027635 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:17.027595 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:18:17.027635 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:17.027605 2557 projected.go:196] Error preparing data for projected volume kube-api-access-hx2b9 for pod openshift-network-diagnostics/network-check-target-kz9ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:17.027720 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:17.027668 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9 podName:3d82edaa-902a-4dd4-a5f3-acb327cfb733 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:49.027651262 +0000 UTC m=+65.390853839 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hx2b9" (UniqueName: "kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9") pod "network-check-target-kz9ls" (UID: "3d82edaa-902a-4dd4-a5f3-acb327cfb733") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:17.194719 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:17.194626 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:17.194719 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:17.194651 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:17.194904 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:17.194752 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wz8mb" podUID="06baddfd-0455-4b6d-8e02-c9fef640b572" Jul 07 02:18:17.194904 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:17.194804 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:17.194904 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:17.194879 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kz9ls" podUID="3d82edaa-902a-4dd4-a5f3-acb327cfb733" Jul 07 02:18:17.194991 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:17.194973 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4sg5" podUID="2379c48f-d209-40e9-88da-f7427e4e9a18" Jul 07 02:18:17.432313 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:17.432283 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc14b32a-df85-420d-a697-67612b30ac22" containerID="fdf486dd7a7b07692d11e88003bfba1b6766821d8d3a5e19278565629cb349b7" exitCode=0 Jul 07 02:18:17.432731 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:17.432330 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerDied","Data":"fdf486dd7a7b07692d11e88003bfba1b6766821d8d3a5e19278565629cb349b7"} Jul 07 02:18:18.906637 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.906610 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-74.ec2.internal" event="NodeReady" Jul 07 02:18:18.906988 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.906739 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 02:18:18.945554 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.945516 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c"] Jul 07 02:18:18.948985 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.948942 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5"] Jul 07 02:18:18.949172 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.949149 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:18.951329 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.951308 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-94vfd"] Jul 07 02:18:18.951565 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.951539 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:18.953285 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.953245 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw"] Jul 07 02:18:18.953435 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.953330 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:18.954119 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.953836 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 02:18:18.954119 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.954003 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.954119 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.954016 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.954351 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.954248 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fnv6g\"" Jul 07 02:18:18.955228 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.955209 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v"] Jul 07 02:18:18.955354 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.955338 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:18.956219 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.956199 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.956466 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.956448 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 02:18:18.956550 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.956537 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 02:18:18.956611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.956540 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 02:18:18.956661 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.956621 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.956704 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.956687 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wc7pz\"" Jul 07 02:18:18.956955 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.956942 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj"] Jul 07 02:18:18.957068 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.957055 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:18.958652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.958632 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-kxtqh"] Jul 07 02:18:18.958784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.958759 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" Jul 07 02:18:18.960362 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960343 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 02:18:18.960477 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960367 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 02:18:18.960477 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960398 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 02:18:18.960630 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960572 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.960630 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960580 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 02:18:18.960753 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960629 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:18.960753 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960574 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7544685bd5-x8rbm"] Jul 07 02:18:18.960854 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960767 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.960854 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960783 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 02:18:18.960854 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.960826 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.961313 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.961293 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.961415 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.961331 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.961415 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.961336 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.961517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.961504 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2k7p9\"" Jul 07 02:18:18.962765 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.962358 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qxr5d\"" Jul 07 02:18:18.962765 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.962669 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gzckg\"" Jul 07 02:18:18.962927 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.962791 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7bbd949cd7-zhfd6"] Jul 07 02:18:18.963032 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.963007 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.963470 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.963242 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.963470 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.963327 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:18.969805 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.968016 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 02:18:18.969805 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.968268 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.969805 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.968305 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 02:18:18.969805 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.968764 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 02:18:18.969805 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.969461 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 02:18:18.971019 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.970996 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xvpsz\"" Jul 07 02:18:18.971019 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.971015 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.974996 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.972878 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-56fvm\"" Jul 07 02:18:18.974996 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.974651 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:18.975251 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.975054 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 02:18:18.975251 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.975237 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5"] Jul 07 02:18:18.975937 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.975906 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-94vfd"] Jul 07 02:18:18.976983 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.976962 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj"] Jul 07 02:18:18.978945 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.978926 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw"] Jul 07 02:18:18.979588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.979568 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-kxtqh"] Jul 07 02:18:18.980445 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.980419 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.980616 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.980593 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-96d87"] Jul 07 02:18:18.980895 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.980878 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kxf76\"" Jul 07 02:18:18.982744 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.981852 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ncxp5\"" Jul 07 02:18:18.983750 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.983719 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" Jul 07 02:18:18.984454 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.984405 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 02:18:18.984561 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.984532 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c"] Jul 07 02:18:18.984904 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.984885 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.985040 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.985025 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 02:18:18.985164 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.985141 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 02:18:18.985568 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.985547 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7544685bd5-x8rbm"] Jul 07 02:18:18.985661 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.985623 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 02:18:18.986843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.986758 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 02:18:18.986843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.986813 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 02:18:18.987105 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.986961 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 02:18:18.989095 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.989074 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 02:18:18.989520 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.989501 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 02:18:18.989616 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.989565 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5dhrj\"" Jul 07 02:18:18.992243 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.992088 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v"] Jul 07 02:18:18.992990 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.992965 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-96d87"] Jul 07 02:18:18.999131 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:18.998918 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7bbd949cd7-zhfd6"] Jul 07 02:18:19.008698 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.008673 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c7gd2"] Jul 07 02:18:19.015687 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.015659 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8nsjg"] Jul 07 02:18:19.015817 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.015782 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.017921 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.017899 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.019099 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.019032 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4v9ss\"" Jul 07 02:18:19.019219 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.019146 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 02:18:19.019561 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.019539 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 02:18:19.020926 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.020906 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 02:18:19.021064 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.021048 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 02:18:19.021129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.021072 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tp7rb\"" Jul 07 02:18:19.021129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.020975 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 02:18:19.027012 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.026990 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8nsjg"] Jul 07 02:18:19.030828 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.030804 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c7gd2"] Jul 07 02:18:19.041576 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041551 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mqf\" (UniqueName: \"kubernetes.io/projected/7f477398-c6db-427b-bca8-45d3675251e9-kube-api-access-86mqf\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:19.041681 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041592 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-serving-cert\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.041681 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9660f59c-3137-4943-bcc2-a94a66b279fd-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.041681 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041653 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-default-certificate\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.041798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041681 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.041798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041708 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9660f59c-3137-4943-bcc2-a94a66b279fd-config\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.041798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041761 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:19.041798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041792 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958hb\" (UniqueName: \"kubernetes.io/projected/2841fb76-869e-4fdb-8394-38d9432dea0e-kube-api-access-958hb\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.041979 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041860 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-stats-auth\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.041979 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041925 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.041979 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxwb\" (UniqueName: \"kubernetes.io/projected/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-kube-api-access-vlxwb\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.042196 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.041985 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8be49692-e2f1-4968-b6e3-b5993acaeb52-ca-trust-extracted\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.042196 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042011 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-config\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.042196 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042056 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-trusted-ca\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.042196 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042095 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.042196 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042148 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpf9z\" (UniqueName: \"kubernetes.io/projected/9660f59c-3137-4943-bcc2-a94a66b279fd-kube-api-access-bpf9z\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.042426 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042206 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjd92\" (UniqueName: \"kubernetes.io/projected/647ff255-7388-4f76-8076-fefe3bc177b8-kube-api-access-tjd92\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.042426 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042260 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-trusted-ca\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.042426 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042310 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-bound-sa-token\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.042426 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042358 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrlh\" (UniqueName: \"kubernetes.io/projected/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-kube-api-access-khrlh\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.042426 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042402 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/647ff255-7388-4f76-8076-fefe3bc177b8-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.042426 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-tmp\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-image-registry-private-configuration\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9pz2\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-kube-api-access-b9pz2\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042505 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042545 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-certificates\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042568 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrhd\" (UniqueName: \"kubernetes.io/projected/714d46ff-0d58-4ab4-861d-93a62727db76-kube-api-access-cxrhd\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.042649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042605 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2841fb76-869e-4fdb-8394-38d9432dea0e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.043016 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042703 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-snapshots\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.043016 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042740 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/647ff255-7388-4f76-8076-fefe3bc177b8-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.043016 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042768 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-installation-pull-secrets\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.043016 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4288\" (UniqueName: \"kubernetes.io/projected/b2e9226c-dfb9-4341-8138-3ab4af00e3c3-kube-api-access-g4288\") pod \"volume-data-source-validator-5d456fdbcd-rzrfj\" (UID: \"b2e9226c-dfb9-4341-8138-3ab4af00e3c3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" Jul 07 02:18:19.043016 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.042829 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-serving-cert\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.143190 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143145 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-certificates\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.143190 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143184 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrhd\" (UniqueName: \"kubernetes.io/projected/714d46ff-0d58-4ab4-861d-93a62727db76-kube-api-access-cxrhd\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.143434 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2841fb76-869e-4fdb-8394-38d9432dea0e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.143434 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-snapshots\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.143434 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143412 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/647ff255-7388-4f76-8076-fefe3bc177b8-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.143611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143445 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggdh\" (UniqueName: \"kubernetes.io/projected/aa97054e-2ebe-4769-9e9a-09526631f660-kube-api-access-5ggdh\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.143611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-installation-pull-secrets\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.143611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143502 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4288\" (UniqueName: \"kubernetes.io/projected/b2e9226c-dfb9-4341-8138-3ab4af00e3c3-kube-api-access-g4288\") pod \"volume-data-source-validator-5d456fdbcd-rzrfj\" (UID: \"b2e9226c-dfb9-4341-8138-3ab4af00e3c3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" Jul 07 02:18:19.143611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143534 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-serving-cert\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.143611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mqf\" (UniqueName: \"kubernetes.io/projected/7f477398-c6db-427b-bca8-45d3675251e9-kube-api-access-86mqf\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143614 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-serving-cert\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9660f59c-3137-4943-bcc2-a94a66b279fd-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143665 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-default-certificate\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143690 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143713 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9660f59c-3137-4943-bcc2-a94a66b279fd-config\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143751 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143786 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5d5655-53e3-4359-adde-5e1da624dd44-cert\") pod \"ingress-canary-8nsjg\" (UID: \"2b5d5655-53e3-4359-adde-5e1da624dd44\") " pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143816 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-958hb\" (UniqueName: \"kubernetes.io/projected/2841fb76-869e-4fdb-8394-38d9432dea0e-kube-api-access-958hb\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143843 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-stats-auth\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.143868 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143858 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-certificates\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143881 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143910 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxwb\" (UniqueName: \"kubernetes.io/projected/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-kube-api-access-vlxwb\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143934 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-snapshots\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.143969 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143999 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/647ff255-7388-4f76-8076-fefe3bc177b8-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.144033 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls podName:7f477398-c6db-427b-bca8-45d3675251e9 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:19.644013434 +0000 UTC m=+36.007216006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-w8v9v" (UID: "7f477398-c6db-427b-bca8-45d3675251e9") : secret "samples-operator-tls" not found Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.144041 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.144102 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls podName:2841fb76-869e-4fdb-8394-38d9432dea0e nodeName:}" failed. No retries permitted until 2026-07-07 02:18:19.644086038 +0000 UTC m=+36.007288601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8wm6c" (UID: "2841fb76-869e-4fdb-8394-38d9432dea0e") : secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144141 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2841fb76-869e-4fdb-8394-38d9432dea0e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.143936 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprtj\" (UniqueName: \"kubernetes.io/projected/2b5d5655-53e3-4359-adde-5e1da624dd44-kube-api-access-xprtj\") pod \"ingress-canary-8nsjg\" (UID: \"2b5d5655-53e3-4359-adde-5e1da624dd44\") " pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144211 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8be49692-e2f1-4968-b6e3-b5993acaeb52-ca-trust-extracted\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa97054e-2ebe-4769-9e9a-09526631f660-config-volume\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144269 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa97054e-2ebe-4769-9e9a-09526631f660-tmp-dir\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144298 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-config\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144323 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-trusted-ca\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.144340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144403 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpf9z\" (UniqueName: \"kubernetes.io/projected/9660f59c-3137-4943-bcc2-a94a66b279fd-kube-api-access-bpf9z\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144434 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjd92\" (UniqueName: \"kubernetes.io/projected/647ff255-7388-4f76-8076-fefe3bc177b8-kube-api-access-tjd92\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144460 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-trusted-ca\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144491 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhth\" (UniqueName: \"kubernetes.io/projected/70ffaf41-0b91-441b-a263-dce2dfa138af-kube-api-access-zkhth\") pod \"network-check-source-989f8b8dc-96d87\" (UID: \"70ffaf41-0b91-441b-a263-dce2dfa138af\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144525 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9660f59c-3137-4943-bcc2-a94a66b279fd-config\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144528 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-bound-sa-token\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144577 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrlh\" (UniqueName: \"kubernetes.io/projected/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-kube-api-access-khrlh\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144607 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/647ff255-7388-4f76-8076-fefe3bc177b8-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144631 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-tmp\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144657 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-image-registry-private-configuration\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144685 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9pz2\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-kube-api-access-b9pz2\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.144708 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.144969 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 02:18:19.145130 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.145031 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:19.645016841 +0000 UTC m=+36.008219424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : secret "router-metrics-certs-default" not found Jul 07 02:18:19.145794 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.145167 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.145794 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.145642 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8be49692-e2f1-4968-b6e3-b5993acaeb52-ca-trust-extracted\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.146125 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-trusted-ca\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.146227 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:19.646213439 +0000 UTC m=+36.009416003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : configmap references non-existent config key: service-ca.crt Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.147407 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-trusted-ca\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.147473 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.147477 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-tmp\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.147505 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.147533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.147664 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.147677 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7544685bd5-x8rbm: secret "image-registry-tls" not found Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.147716 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls podName:8be49692-e2f1-4968-b6e3-b5993acaeb52 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:19.647703211 +0000 UTC m=+36.010905775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls") pod "image-registry-7544685bd5-x8rbm" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52") : secret "image-registry-tls" not found Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.147889 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-config\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.148070 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.147993 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.149997 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.149952 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-serving-cert\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.150127 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.150102 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-stats-auth\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.150195 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.150133 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-serving-cert\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.150434 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.150410 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9660f59c-3137-4943-bcc2-a94a66b279fd-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.151448 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.151424 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-default-certificate\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.151819 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.151797 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-image-registry-private-configuration\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.152123 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.152096 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-installation-pull-secrets\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.152260 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.152235 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/647ff255-7388-4f76-8076-fefe3bc177b8-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.163102 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.163073 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrhd\" (UniqueName: \"kubernetes.io/projected/714d46ff-0d58-4ab4-861d-93a62727db76-kube-api-access-cxrhd\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.163904 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.163878 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4288\" (UniqueName: \"kubernetes.io/projected/b2e9226c-dfb9-4341-8138-3ab4af00e3c3-kube-api-access-g4288\") pod \"volume-data-source-validator-5d456fdbcd-rzrfj\" (UID: \"b2e9226c-dfb9-4341-8138-3ab4af00e3c3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" Jul 07 02:18:19.166586 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.166559 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mqf\" (UniqueName: \"kubernetes.io/projected/7f477398-c6db-427b-bca8-45d3675251e9-kube-api-access-86mqf\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:19.167517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.167491 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-bound-sa-token\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.167963 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.167925 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpf9z\" (UniqueName: \"kubernetes.io/projected/9660f59c-3137-4943-bcc2-a94a66b279fd-kube-api-access-bpf9z\") pod \"service-ca-operator-5d46d9ffd4-xc9b5\" (UID: \"9660f59c-3137-4943-bcc2-a94a66b279fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.169936 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.169901 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrlh\" (UniqueName: \"kubernetes.io/projected/ba3f3d85-b305-4bb7-a031-640fc9f4f2e3-kube-api-access-khrlh\") pod \"insights-operator-5c847dcfc4-94vfd\" (UID: \"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3\") " pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.171905 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.171880 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxwb\" (UniqueName: \"kubernetes.io/projected/67e6975a-a327-4c7f-b7ce-23ab9bc3d059-kube-api-access-vlxwb\") pod \"console-operator-68b757fb67-kxtqh\" (UID: \"67e6975a-a327-4c7f-b7ce-23ab9bc3d059\") " pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.173178 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.173149 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjd92\" (UniqueName: \"kubernetes.io/projected/647ff255-7388-4f76-8076-fefe3bc177b8-kube-api-access-tjd92\") pod \"kube-storage-version-migrator-operator-846bf668f7-7l7pw\" (UID: \"647ff255-7388-4f76-8076-fefe3bc177b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.174578 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.174541 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958hb\" (UniqueName: \"kubernetes.io/projected/2841fb76-869e-4fdb-8394-38d9432dea0e-kube-api-access-958hb\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.178922 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.178874 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9pz2\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-kube-api-access-b9pz2\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.194627 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.194596 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:19.194627 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.194611 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:19.194842 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.194603 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:19.201750 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.201721 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 02:18:19.202330 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.201736 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv29t\"" Jul 07 02:18:19.202625 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.201761 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kthkt\"" Jul 07 02:18:19.205117 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.205097 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 02:18:19.248242 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248204 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5d5655-53e3-4359-adde-5e1da624dd44-cert\") pod \"ingress-canary-8nsjg\" (UID: \"2b5d5655-53e3-4359-adde-5e1da624dd44\") " pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.248517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprtj\" (UniqueName: \"kubernetes.io/projected/2b5d5655-53e3-4359-adde-5e1da624dd44-kube-api-access-xprtj\") pod \"ingress-canary-8nsjg\" (UID: \"2b5d5655-53e3-4359-adde-5e1da624dd44\") " pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.248517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248278 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa97054e-2ebe-4769-9e9a-09526631f660-config-volume\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.248517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248303 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa97054e-2ebe-4769-9e9a-09526631f660-tmp-dir\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.248517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248344 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhth\" (UniqueName: \"kubernetes.io/projected/70ffaf41-0b91-441b-a263-dce2dfa138af-kube-api-access-zkhth\") pod \"network-check-source-989f8b8dc-96d87\" (UID: \"70ffaf41-0b91-441b-a263-dce2dfa138af\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" Jul 07 02:18:19.248517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248421 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.248517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggdh\" (UniqueName: \"kubernetes.io/projected/aa97054e-2ebe-4769-9e9a-09526631f660-kube-api-access-5ggdh\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.248819 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.248554 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:19.248819 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.248627 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls podName:aa97054e-2ebe-4769-9e9a-09526631f660 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:19.748608543 +0000 UTC m=+36.111811125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls") pod "dns-default-c7gd2" (UID: "aa97054e-2ebe-4769-9e9a-09526631f660") : secret "dns-default-metrics-tls" not found Jul 07 02:18:19.248925 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.248819 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa97054e-2ebe-4769-9e9a-09526631f660-tmp-dir\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.249063 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.249031 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa97054e-2ebe-4769-9e9a-09526631f660-config-volume\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.250815 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.250795 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5d5655-53e3-4359-adde-5e1da624dd44-cert\") pod \"ingress-canary-8nsjg\" (UID: \"2b5d5655-53e3-4359-adde-5e1da624dd44\") " pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.270612 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.270579 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggdh\" (UniqueName: \"kubernetes.io/projected/aa97054e-2ebe-4769-9e9a-09526631f660-kube-api-access-5ggdh\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.270751 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.270643 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprtj\" (UniqueName: \"kubernetes.io/projected/2b5d5655-53e3-4359-adde-5e1da624dd44-kube-api-access-xprtj\") pod \"ingress-canary-8nsjg\" (UID: \"2b5d5655-53e3-4359-adde-5e1da624dd44\") " pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.270751 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.270655 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhth\" (UniqueName: \"kubernetes.io/projected/70ffaf41-0b91-441b-a263-dce2dfa138af-kube-api-access-zkhth\") pod \"network-check-source-989f8b8dc-96d87\" (UID: \"70ffaf41-0b91-441b-a263-dce2dfa138af\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" Jul 07 02:18:19.278482 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.278445 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" Jul 07 02:18:19.289491 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.289461 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" Jul 07 02:18:19.295262 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.295235 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:19.299144 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.299113 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" Jul 07 02:18:19.316502 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.316461 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" Jul 07 02:18:19.326498 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.326169 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:19.347668 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.347360 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" Jul 07 02:18:19.362074 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.360820 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8nsjg" Jul 07 02:18:19.413628 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.413389 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:19.545474 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.544789 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-94vfd"] Jul 07 02:18:19.547203 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.547116 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5"] Jul 07 02:18:19.565731 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.565424 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw"] Jul 07 02:18:19.568483 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:19.568454 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647ff255_7388_4f76_8076_fefe3bc177b8.slice/crio-557b6da5ff6e2f05a60b629ce4dec713939bd3240631ccf6fae4ee84ae97cf22 WatchSource:0}: Error finding container 557b6da5ff6e2f05a60b629ce4dec713939bd3240631ccf6fae4ee84ae97cf22: Status 404 returned error can't find the container with id 557b6da5ff6e2f05a60b629ce4dec713939bd3240631ccf6fae4ee84ae97cf22 Jul 07 02:18:19.582662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.582627 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-kxtqh"] Jul 07 02:18:19.587111 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:19.587077 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e6975a_a327_4c7f_b7ce_23ab9bc3d059.slice/crio-d5a8ba6a1e09f76abf9a69cb85a657f62806ad5d87ad7c53e71de500e404be7a WatchSource:0}: Error finding container d5a8ba6a1e09f76abf9a69cb85a657f62806ad5d87ad7c53e71de500e404be7a: Status 404 returned error can't find the container with id d5a8ba6a1e09f76abf9a69cb85a657f62806ad5d87ad7c53e71de500e404be7a Jul 07 02:18:19.596125 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.596095 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj"] Jul 07 02:18:19.599962 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:19.599929 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e9226c_dfb9_4341_8138_3ab4af00e3c3.slice/crio-73a9da22de70664fab762a60a1a89fbca71cb3acae0fd80099d68017939a2462 WatchSource:0}: Error finding container 73a9da22de70664fab762a60a1a89fbca71cb3acae0fd80099d68017939a2462: Status 404 returned error can't find the container with id 73a9da22de70664fab762a60a1a89fbca71cb3acae0fd80099d68017939a2462 Jul 07 02:18:19.611164 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.611143 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-96d87"] Jul 07 02:18:19.614704 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:19.614675 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ffaf41_0b91_441b_a263_dce2dfa138af.slice/crio-20d0c8d72829ffb75016d2c1a7d1b37326f8764628561e258a0dacdf3add2736 WatchSource:0}: Error finding container 20d0c8d72829ffb75016d2c1a7d1b37326f8764628561e258a0dacdf3add2736: Status 404 returned error can't find the container with id 20d0c8d72829ffb75016d2c1a7d1b37326f8764628561e258a0dacdf3add2736 Jul 07 02:18:19.618045 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.618021 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8nsjg"] Jul 07 02:18:19.622359 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:19.622331 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5d5655_53e3_4359_adde_5e1da624dd44.slice/crio-2c79c089e6537e182b25b5da441288665eb987e8f20bd62513b14c1ce44efefa WatchSource:0}: Error finding container 2c79c089e6537e182b25b5da441288665eb987e8f20bd62513b14c1ce44efefa: Status 404 returned error can't find the container with id 2c79c089e6537e182b25b5da441288665eb987e8f20bd62513b14c1ce44efefa Jul 07 02:18:19.655832 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.655802 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.655992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.655861 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:19.655992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.655904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:19.656122 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.655988 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 02:18:19.656122 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.655983 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:19.656122 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656035 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:18:19.656122 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.656044 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:19.656122 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656054 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7544685bd5-x8rbm: secret "image-registry-tls" not found Jul 07 02:18:19.656122 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656065 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.656044515 +0000 UTC m=+37.019247096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : secret "router-metrics-certs-default" not found Jul 07 02:18:19.656122 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656106 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls podName:8be49692-e2f1-4968-b6e3-b5993acaeb52 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.656089622 +0000 UTC m=+37.019292197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls") pod "image-registry-7544685bd5-x8rbm" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52") : secret "image-registry-tls" not found Jul 07 02:18:19.656399 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656143 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 02:18:19.656399 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656153 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:19.656399 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656156 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.656138892 +0000 UTC m=+37.019341457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : configmap references non-existent config key: service-ca.crt Jul 07 02:18:19.656399 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656209 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls podName:7f477398-c6db-427b-bca8-45d3675251e9 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.656191877 +0000 UTC m=+37.019394457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-w8v9v" (UID: "7f477398-c6db-427b-bca8-45d3675251e9") : secret "samples-operator-tls" not found Jul 07 02:18:19.656399 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.656237 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls podName:2841fb76-869e-4fdb-8394-38d9432dea0e nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.656227384 +0000 UTC m=+37.019429946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8wm6c" (UID: "2841fb76-869e-4fdb-8394-38d9432dea0e") : secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:19.756820 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:19.756771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:19.757010 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.756944 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:19.757074 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:19.757019 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls podName:aa97054e-2ebe-4769-9e9a-09526631f660 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.756998676 +0000 UTC m=+37.120201262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls") pod "dns-default-c7gd2" (UID: "aa97054e-2ebe-4769-9e9a-09526631f660") : secret "dns-default-metrics-tls" not found Jul 07 02:18:20.453927 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.453885 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8nsjg" event={"ID":"2b5d5655-53e3-4359-adde-5e1da624dd44","Type":"ContainerStarted","Data":"2c79c089e6537e182b25b5da441288665eb987e8f20bd62513b14c1ce44efefa"} Jul 07 02:18:20.465097 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.464998 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" event={"ID":"70ffaf41-0b91-441b-a263-dce2dfa138af","Type":"ContainerStarted","Data":"20d0c8d72829ffb75016d2c1a7d1b37326f8764628561e258a0dacdf3add2736"} Jul 07 02:18:20.469478 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.469407 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" event={"ID":"67e6975a-a327-4c7f-b7ce-23ab9bc3d059","Type":"ContainerStarted","Data":"d5a8ba6a1e09f76abf9a69cb85a657f62806ad5d87ad7c53e71de500e404be7a"} Jul 07 02:18:20.472086 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.472008 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" event={"ID":"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3","Type":"ContainerStarted","Data":"c2b6072739afebcc26fb878427c56b61f2239d401cd36e4393bab58c3bed66a4"} Jul 07 02:18:20.476549 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.476468 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" event={"ID":"647ff255-7388-4f76-8076-fefe3bc177b8","Type":"ContainerStarted","Data":"557b6da5ff6e2f05a60b629ce4dec713939bd3240631ccf6fae4ee84ae97cf22"} Jul 07 02:18:20.487403 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.486990 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" event={"ID":"b2e9226c-dfb9-4341-8138-3ab4af00e3c3","Type":"ContainerStarted","Data":"73a9da22de70664fab762a60a1a89fbca71cb3acae0fd80099d68017939a2462"} Jul 07 02:18:20.492536 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.490562 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" event={"ID":"9660f59c-3137-4943-bcc2-a94a66b279fd","Type":"ContainerStarted","Data":"349769cf5503d588eff5c7ecd1f92cc1106cb031b3c8ffc883850d3a462cd356"} Jul 07 02:18:20.664168 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.664127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:20.664338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.664196 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:20.664338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.664229 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:20.664338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.664296 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:20.664529 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.664336 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:20.664529 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.664504 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 02:18:20.664621 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.664574 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls podName:7f477398-c6db-427b-bca8-45d3675251e9 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:22.664553739 +0000 UTC m=+39.027756320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-w8v9v" (UID: "7f477398-c6db-427b-bca8-45d3675251e9") : secret "samples-operator-tls" not found Jul 07 02:18:20.664699 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.664683 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:22.664649359 +0000 UTC m=+39.027851926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : configmap references non-existent config key: service-ca.crt Jul 07 02:18:20.664847 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.664831 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 02:18:20.664923 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.664906 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:22.664892632 +0000 UTC m=+39.028095208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : secret "router-metrics-certs-default" not found Jul 07 02:18:20.664998 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.664985 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:18:20.665054 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.665025 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7544685bd5-x8rbm: secret "image-registry-tls" not found Jul 07 02:18:20.665104 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.665064 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls podName:8be49692-e2f1-4968-b6e3-b5993acaeb52 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:22.665051557 +0000 UTC m=+39.028254133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls") pod "image-registry-7544685bd5-x8rbm" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52") : secret "image-registry-tls" not found Jul 07 02:18:20.665165 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.665131 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:20.665213 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.665198 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls podName:2841fb76-869e-4fdb-8394-38d9432dea0e nodeName:}" failed. No retries permitted until 2026-07-07 02:18:22.665186265 +0000 UTC m=+39.028388842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8wm6c" (UID: "2841fb76-869e-4fdb-8394-38d9432dea0e") : secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:20.766039 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:20.765156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:20.766039 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.765613 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:20.766039 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:20.765691 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls podName:aa97054e-2ebe-4769-9e9a-09526631f660 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:22.765671904 +0000 UTC m=+39.128874488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls") pod "dns-default-c7gd2" (UID: "aa97054e-2ebe-4769-9e9a-09526631f660") : secret "dns-default-metrics-tls" not found Jul 07 02:18:21.724261 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:21.724223 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.683994 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.684048 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.684132 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684149 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.684174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684215 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.68419648 +0000 UTC m=+43.047399042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : secret "router-metrics-certs-default" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684270 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684308 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls podName:7f477398-c6db-427b-bca8-45d3675251e9 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.684296721 +0000 UTC m=+43.047499298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-w8v9v" (UID: "7f477398-c6db-427b-bca8-45d3675251e9") : secret "samples-operator-tls" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.684267 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684334 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.684325462 +0000 UTC m=+43.047528027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : configmap references non-existent config key: service-ca.crt Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684453 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684465 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7544685bd5-x8rbm: secret "image-registry-tls" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684499 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls podName:8be49692-e2f1-4968-b6e3-b5993acaeb52 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.684488818 +0000 UTC m=+43.047691395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls") pod "image-registry-7544685bd5-x8rbm" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52") : secret "image-registry-tls" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684549 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:22.684605 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.684577 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls podName:2841fb76-869e-4fdb-8394-38d9432dea0e nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.68456793 +0000 UTC m=+43.047770506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8wm6c" (UID: "2841fb76-869e-4fdb-8394-38d9432dea0e") : secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:22.785331 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.785291 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:22.785797 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.785590 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:22.785797 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:22.785669 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls podName:aa97054e-2ebe-4769-9e9a-09526631f660 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:26.785652995 +0000 UTC m=+43.148855557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls") pod "dns-default-c7gd2" (UID: "aa97054e-2ebe-4769-9e9a-09526631f660") : secret "dns-default-metrics-tls" not found Jul 07 02:18:22.924205 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.924151 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:22.985071 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:22.984883 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:23.425274 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:23.425226 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:23.480325 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:23.480280 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:24.203697 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:24.202967 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:24.266438 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:24.266398 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:26.718693 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:26.718651 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:26.718712 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:26.718764 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:26.718809 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718824 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:26.718847 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718891 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls podName:2841fb76-869e-4fdb-8394-38d9432dea0e nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.718872182 +0000 UTC m=+51.082074745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8wm6c" (UID: "2841fb76-869e-4fdb-8394-38d9432dea0e") : secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718914 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718929 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718935 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.718913928 +0000 UTC m=+51.082116489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : configmap references non-existent config key: service-ca.crt Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718941 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7544685bd5-x8rbm: secret "image-registry-tls" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718972 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls podName:7f477398-c6db-427b-bca8-45d3675251e9 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.718956331 +0000 UTC m=+51.082158919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-w8v9v" (UID: "7f477398-c6db-427b-bca8-45d3675251e9") : secret "samples-operator-tls" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.718998 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.719007 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls podName:8be49692-e2f1-4968-b6e3-b5993acaeb52 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.718995036 +0000 UTC m=+51.082197598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls") pod "image-registry-7544685bd5-x8rbm" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52") : secret "image-registry-tls" not found Jul 07 02:18:26.719219 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.719059 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.719045701 +0000 UTC m=+51.082248267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : secret "router-metrics-certs-default" not found Jul 07 02:18:26.819738 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:26.819696 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:26.819923 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.819868 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:26.819996 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:26.819986 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls podName:aa97054e-2ebe-4769-9e9a-09526631f660 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.819965617 +0000 UTC m=+51.183168179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls") pod "dns-default-c7gd2" (UID: "aa97054e-2ebe-4769-9e9a-09526631f660") : secret "dns-default-metrics-tls" not found Jul 07 02:18:29.360963 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:29.360937 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:29.462128 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:29.462099 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:30.522726 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.522687 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" event={"ID":"b2e9226c-dfb9-4341-8138-3ab4af00e3c3","Type":"ContainerStarted","Data":"a1a3768b4f9a1f3fff08318e0a91f4f01b88850bbe3e05bb416717b272cee0af"} Jul 07 02:18:30.524269 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.524236 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" event={"ID":"9660f59c-3137-4943-bcc2-a94a66b279fd","Type":"ContainerStarted","Data":"27a02d5959329e1889f2807d1fde61a2cb4d71dca0e967a35c35adbcdad3b803"} Jul 07 02:18:30.527506 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.527479 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc14b32a-df85-420d-a697-67612b30ac22" containerID="02a31a5381bcb9d159998d47b34b7a36f1ff2df2ace604e650074153aee1821e" exitCode=0 Jul 07 02:18:30.527633 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.527553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerDied","Data":"02a31a5381bcb9d159998d47b34b7a36f1ff2df2ace604e650074153aee1821e"} Jul 07 02:18:30.529835 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.529815 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8nsjg" event={"ID":"2b5d5655-53e3-4359-adde-5e1da624dd44","Type":"ContainerStarted","Data":"2cbb44940888c58b34b47d3dc62c1727ebfe2f843b8b8873a965eec8a628e1dc"} Jul 07 02:18:30.532754 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.532736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" event={"ID":"70ffaf41-0b91-441b-a263-dce2dfa138af","Type":"ContainerStarted","Data":"7a75d80e936e2263c6940b12a207cd19f95a6075697ff5d708b6adad23bcdc51"} Jul 07 02:18:30.534881 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.534861 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/0.log" Jul 07 02:18:30.534980 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.534895 2557 generic.go:356] "Generic (PLEG): container finished" podID="67e6975a-a327-4c7f-b7ce-23ab9bc3d059" containerID="b5d42fa7892e43833c6682ed1904d1d178a0fc1e835964c2a016ebc19ede2112" exitCode=255 Jul 07 02:18:30.534980 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.534962 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" event={"ID":"67e6975a-a327-4c7f-b7ce-23ab9bc3d059","Type":"ContainerDied","Data":"b5d42fa7892e43833c6682ed1904d1d178a0fc1e835964c2a016ebc19ede2112"} Jul 07 02:18:30.535290 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.535257 2557 scope.go:117] "RemoveContainer" containerID="b5d42fa7892e43833c6682ed1904d1d178a0fc1e835964c2a016ebc19ede2112" Jul 07 02:18:30.536910 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.536827 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" event={"ID":"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3","Type":"ContainerStarted","Data":"480bf0271ee8f6884074e291895fe537f1cb0e4f1200a7903b63454675fd36ec"} Jul 07 02:18:30.538472 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.538400 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" event={"ID":"647ff255-7388-4f76-8076-fefe3bc177b8","Type":"ContainerStarted","Data":"5500f66a80a74596126ae8257a22b4b1769ce4797aeb07fa606d4bff7863fe3c"} Jul 07 02:18:30.539928 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.539882 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rzrfj" podStartSLOduration=11.739504761 podStartE2EDuration="21.539866465s" totalImagesPullingTime="9.800361704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.539017669 +0000 UTC m=+46.902220251" watchObservedRunningTime="2026-07-07 02:18:30.539866465 +0000 UTC m=+46.903069028" Jul 07 02:18:30.573746 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.573696 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-96d87" podStartSLOduration=9.787399171 podStartE2EDuration="19.573676045s" totalImagesPullingTime="9.786276874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.572447969 +0000 UTC m=+46.935650555" watchObservedRunningTime="2026-07-07 02:18:30.573676045 +0000 UTC m=+46.936878630" Jul 07 02:18:30.593936 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.593084 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" podStartSLOduration=11.752367813 podStartE2EDuration="21.593064649s" totalImagesPullingTime="9.840696836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.59204553 +0000 UTC m=+46.955248138" watchObservedRunningTime="2026-07-07 02:18:30.593064649 +0000 UTC m=+46.956267234" Jul 07 02:18:30.615357 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.613432 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8nsjg" podStartSLOduration=2.824960238 podStartE2EDuration="12.613413253s" totalImagesPullingTime="9.788453015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.612554893 +0000 UTC m=+46.975757479" watchObservedRunningTime="2026-07-07 02:18:30.613413253 +0000 UTC m=+46.976615838" Jul 07 02:18:30.636275 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.636251 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:30.669865 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.669810 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" podStartSLOduration=11.822782653 podStartE2EDuration="21.669765493s" totalImagesPullingTime="9.84698284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.66920097 +0000 UTC m=+47.032403555" watchObservedRunningTime="2026-07-07 02:18:30.669765493 +0000 UTC m=+47.032968080" Jul 07 02:18:30.692706 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.692244 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" podStartSLOduration=11.856819039 podStartE2EDuration="21.692223938s" totalImagesPullingTime="9.835404899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.691633472 +0000 UTC m=+47.054836056" watchObservedRunningTime="2026-07-07 02:18:30.692223938 +0000 UTC m=+47.055426503" Jul 07 02:18:30.931568 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:30.931525 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.104568 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.104532 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.203098 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.203066 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sh5gt"] Jul 07 02:18:31.217349 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.217317 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.220796 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.220768 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 02:18:31.222949 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.222929 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vjbfk\"" Jul 07 02:18:31.223580 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.223553 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 02:18:31.232394 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.232343 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sh5gt"] Jul 07 02:18:31.255460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.255427 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.271801 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.271769 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5qrq\" (UniqueName: \"kubernetes.io/projected/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-api-access-t5qrq\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.271964 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.271820 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fa37f6f-940a-48bc-91bf-e04f86947ca5-data-volume\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.271964 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.271884 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.272048 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.271987 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.272048 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.272016 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fa37f6f-940a-48bc-91bf-e04f86947ca5-crio-socket\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.350110 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.350070 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.372817 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.372736 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qrq\" (UniqueName: \"kubernetes.io/projected/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-api-access-t5qrq\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.372817 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.372782 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fa37f6f-940a-48bc-91bf-e04f86947ca5-data-volume\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.373021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.372819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.373021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.372897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.373021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.372914 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fa37f6f-940a-48bc-91bf-e04f86947ca5-crio-socket\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.373164 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.373019 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:18:31.373164 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.373031 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:18:31.373164 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.373059 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fa37f6f-940a-48bc-91bf-e04f86947ca5-crio-socket\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.373164 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.373091 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:31.873072609 +0000 UTC m=+48.236275176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : secret "insights-runtime-extractor-tls" not found Jul 07 02:18:31.373164 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.373112 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:31.873103008 +0000 UTC m=+48.236305573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : configmap "kube-rbac-proxy" not found Jul 07 02:18:31.373417 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.373202 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fa37f6f-940a-48bc-91bf-e04f86947ca5-data-volume\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.382970 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.382951 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qrq\" (UniqueName: \"kubernetes.io/projected/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-api-access-t5qrq\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.420099 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.420043 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.544468 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.544439 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:18:31.544854 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.544838 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/0.log" Jul 07 02:18:31.544903 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.544872 2557 generic.go:356] "Generic (PLEG): container finished" podID="67e6975a-a327-4c7f-b7ce-23ab9bc3d059" containerID="6ac645bf0ba01f2a5b6257648b4e11ca884dc4c13caaea9652a40c7c9e99e315" exitCode=255 Jul 07 02:18:31.544995 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.544974 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" event={"ID":"67e6975a-a327-4c7f-b7ce-23ab9bc3d059","Type":"ContainerDied","Data":"6ac645bf0ba01f2a5b6257648b4e11ca884dc4c13caaea9652a40c7c9e99e315"} Jul 07 02:18:31.545047 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.545027 2557 scope.go:117] "RemoveContainer" containerID="b5d42fa7892e43833c6682ed1904d1d178a0fc1e835964c2a016ebc19ede2112" Jul 07 02:18:31.545265 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.545246 2557 scope.go:117] "RemoveContainer" containerID="6ac645bf0ba01f2a5b6257648b4e11ca884dc4c13caaea9652a40c7c9e99e315" Jul 07 02:18:31.545488 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.545469 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-kxtqh_openshift-console-operator(67e6975a-a327-4c7f-b7ce-23ab9bc3d059)\"" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" podUID="67e6975a-a327-4c7f-b7ce-23ab9bc3d059" Jul 07 02:18:31.548135 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.548112 2557 generic.go:356] "Generic (PLEG): container finished" podID="fc14b32a-df85-420d-a697-67612b30ac22" containerID="19d6d0f17b50da03e8f7b816990b48db10fb69461bb792d823f4b48b195cd486" exitCode=0 Jul 07 02:18:31.548253 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.548158 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerDied","Data":"19d6d0f17b50da03e8f7b816990b48db10fb69461bb792d823f4b48b195cd486"} Jul 07 02:18:31.734642 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.734611 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.835427 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.835398 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.878282 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.878197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.878464 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.878306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:31.878464 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.878336 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:18:31.878464 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.878433 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:32.878411933 +0000 UTC m=+49.241614500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : configmap "kube-rbac-proxy" not found Jul 07 02:18:31.878464 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.878459 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:18:31.878685 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:31.878500 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:32.878488392 +0000 UTC m=+49.241690955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : secret "insights-runtime-extractor-tls" not found Jul 07 02:18:31.910517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:31.910488 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:32.098480 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.098451 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:32.202021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.201945 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:32.295109 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.295077 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:32.555265 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.555229 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" event={"ID":"fc14b32a-df85-420d-a697-67612b30ac22","Type":"ContainerStarted","Data":"2cb19a454cc7446809ba7c400b0f8129eff2de089be478408062105772240058"} Jul 07 02:18:32.556715 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.556695 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:18:32.556976 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.556961 2557 scope.go:117] "RemoveContainer" containerID="6ac645bf0ba01f2a5b6257648b4e11ca884dc4c13caaea9652a40c7c9e99e315" Jul 07 02:18:32.557110 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:32.557094 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-kxtqh_openshift-console-operator(67e6975a-a327-4c7f-b7ce-23ab9bc3d059)\"" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" podUID="67e6975a-a327-4c7f-b7ce-23ab9bc3d059" Jul 07 02:18:32.583986 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.583939 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jf4qg" podStartSLOduration=7.813857851 podStartE2EDuration="48.583923948s" totalImagesPullingTime="40.770066097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:32.581644484 +0000 UTC m=+48.944847080" watchObservedRunningTime="2026-07-07 02:18:32.583923948 +0000 UTC m=+48.947126532" Jul 07 02:18:32.885801 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.885719 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:32.885948 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.885839 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:32.885948 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:32.885845 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:18:32.886017 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:32.885920 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:18:32.886050 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:32.885992 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.885975687 +0000 UTC m=+51.249178250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : configmap "kube-rbac-proxy" not found Jul 07 02:18:32.886050 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:32.886036 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.886025257 +0000 UTC m=+51.249227822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : secret "insights-runtime-extractor-tls" not found Jul 07 02:18:32.958643 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:32.958609 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:33.059078 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:33.059048 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:33.361399 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:33.361326 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:33.426342 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:33.426303 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:33.724110 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:33.724009 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:33.819957 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:33.819932 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z6bxk_6a949319-84f9-4682-9f53-012572cc79ea/dns-node-resolver/0.log" Jul 07 02:18:34.016137 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.016085 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:34.116355 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.116316 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:34.374664 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.374578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:34.419725 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.419702 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cnzk_c68a2cba-9963-440a-96f3-841863503bee/node-ca/0.log" Jul 07 02:18:34.666689 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.666600 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:34.782405 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.782353 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:34.801598 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.801565 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:34.801742 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.801608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:34.801742 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.801671 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:34.801742 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801718 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:34.801742 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.801732 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:34.801918 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.801767 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:34.801918 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801781 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 02:18:34.801918 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801783 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls podName:2841fb76-869e-4fdb-8394-38d9432dea0e nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.801764482 +0000 UTC m=+67.164967059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8wm6c" (UID: "2841fb76-869e-4fdb-8394-38d9432dea0e") : secret "cluster-monitoring-operator-tls" not found Jul 07 02:18:34.801918 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801848 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 02:18:34.801918 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801871 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls podName:7f477398-c6db-427b-bca8-45d3675251e9 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.801853977 +0000 UTC m=+67.165056539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-w8v9v" (UID: "7f477398-c6db-427b-bca8-45d3675251e9") : secret "samples-operator-tls" not found Jul 07 02:18:34.801918 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801905 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:18:34.801918 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801915 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7544685bd5-x8rbm: secret "image-registry-tls" not found Jul 07 02:18:34.802120 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801932 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.801925432 +0000 UTC m=+67.165127994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : secret "router-metrics-certs-default" not found Jul 07 02:18:34.802120 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801944 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle podName:714d46ff-0d58-4ab4-861d-93a62727db76 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.801939037 +0000 UTC m=+67.165141598 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle") pod "router-default-7bbd949cd7-zhfd6" (UID: "714d46ff-0d58-4ab4-861d-93a62727db76") : configmap references non-existent config key: service-ca.crt Jul 07 02:18:34.802120 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.801954 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls podName:8be49692-e2f1-4968-b6e3-b5993acaeb52 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.801949463 +0000 UTC m=+67.165152026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls") pod "image-registry-7544685bd5-x8rbm" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52") : secret "image-registry-tls" not found Jul 07 02:18:34.878551 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.878509 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:34.902707 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.902677 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:34.902847 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.902737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:34.902847 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.902776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:34.902847 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.902819 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:18:34.902952 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.902879 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:38.902864225 +0000 UTC m=+55.266066786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : secret "insights-runtime-extractor-tls" not found Jul 07 02:18:34.902952 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.902819 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:34.902952 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:34.902941 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls podName:aa97054e-2ebe-4769-9e9a-09526631f660 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.902927029 +0000 UTC m=+67.266129592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls") pod "dns-default-c7gd2" (UID: "aa97054e-2ebe-4769-9e9a-09526631f660") : secret "dns-default-metrics-tls" not found Jul 07 02:18:34.903242 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:34.903223 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fa37f6f-940a-48bc-91bf-e04f86947ca5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:35.023703 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:35.023668 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:35.220582 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:35.220546 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8nsjg_2b5d5655-53e3-4359-adde-5e1da624dd44/serve-healthcheck-canary/0.log" Jul 07 02:18:36.022154 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:36.022124 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-7l7pw_647ff255-7388-4f76-8076-fefe3bc177b8/kube-storage-version-migrator-operator/0.log" Jul 07 02:18:37.298850 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:37.298813 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:37.916986 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:37.916951 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:37.970595 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:37.970567 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:38.938457 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:38.938415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:38.938951 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:38.938589 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:18:38.938951 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:38.938680 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls podName:3fa37f6f-940a-48bc-91bf-e04f86947ca5 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:46.938657413 +0000 UTC m=+63.301859995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh5gt" (UID: "3fa37f6f-940a-48bc-91bf-e04f86947ca5") : secret "insights-runtime-extractor-tls" not found Jul 07 02:18:39.068171 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.068142 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:39.268283 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.268236 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:39.327524 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.327484 2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:39.327685 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.327568 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:39.327974 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.327957 2557 scope.go:117] "RemoveContainer" containerID="6ac645bf0ba01f2a5b6257648b4e11ca884dc4c13caaea9652a40c7c9e99e315" Jul 07 02:18:39.328175 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:18:39.328152 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-kxtqh_openshift-console-operator(67e6975a-a327-4c7f-b7ce-23ab9bc3d059)\"" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" podUID="67e6975a-a327-4c7f-b7ce-23ab9bc3d059" Jul 07 02:18:39.487679 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.487636 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:39.592560 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.592466 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:39.700130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.700096 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:39.837458 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:39.837422 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:40.370845 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:40.370812 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:40.943433 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:40.943402 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:41.338887 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:41.338853 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:41.755804 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:41.755773 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:42.141416 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:42.141297 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:42.340302 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:42.340260 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:42.455049 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:42.454967 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:42.871098 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:42.871068 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:42.972214 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:42.972184 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:43.558577 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:43.558547 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:43.749778 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:43.749747 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:43.850666 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:43.850585 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:43.950297 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:43.950262 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:44.346233 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:44.346198 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:44.434671 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:44.434635 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5ll6" Jul 07 02:18:44.743051 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:44.742968 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:45.539307 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:45.539254 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:45.730137 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:45.730106 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:45.784073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:45.784042 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:45.964995 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:45.964915 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:46.091659 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:46.091625 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:46.348748 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:46.348699 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:46.948218 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:46.948185 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:47.011468 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.011442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:47.013735 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.013708 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fa37f6f-940a-48bc-91bf-e04f86947ca5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh5gt\" (UID: \"3fa37f6f-940a-48bc-91bf-e04f86947ca5\") " pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:47.131929 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.131899 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vjbfk\"" Jul 07 02:18:47.140199 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.140183 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sh5gt" Jul 07 02:18:47.262941 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.262911 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sh5gt"] Jul 07 02:18:47.265910 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:47.265885 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa37f6f_940a_48bc_91bf_e04f86947ca5.slice/crio-8a0a8ba66eaa029c78c60e51283d039b849688044950a755d7f1f84b470b60c6 WatchSource:0}: Error finding container 8a0a8ba66eaa029c78c60e51283d039b849688044950a755d7f1f84b470b60c6: Status 404 returned error can't find the container with id 8a0a8ba66eaa029c78c60e51283d039b849688044950a755d7f1f84b470b60c6 Jul 07 02:18:47.544873 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.544830 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:47.604173 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.604142 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh5gt" event={"ID":"3fa37f6f-940a-48bc-91bf-e04f86947ca5","Type":"ContainerStarted","Data":"8a0a8ba66eaa029c78c60e51283d039b849688044950a755d7f1f84b470b60c6"} Jul 07 02:18:47.745213 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.745152 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:47.950132 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:47.949947 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:48.342518 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.342478 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:48.741631 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.741551 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:48.930250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.930216 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:48.930462 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.930299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:48.933223 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.933203 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 02:18:48.934250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.934228 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 02:18:48.943865 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.943839 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2379c48f-d209-40e9-88da-f7427e4e9a18-metrics-certs\") pod \"network-metrics-daemon-q4sg5\" (UID: \"2379c48f-d209-40e9-88da-f7427e4e9a18\") " pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:48.943990 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:48.943946 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06baddfd-0455-4b6d-8e02-c9fef640b572-original-pull-secret\") pod \"global-pull-secret-syncer-wz8mb\" (UID: \"06baddfd-0455-4b6d-8e02-c9fef640b572\") " pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:49.031551 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.031517 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:49.034077 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.034047 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2b9\" (UniqueName: \"kubernetes.io/projected/3d82edaa-902a-4dd4-a5f3-acb327cfb733-kube-api-access-hx2b9\") pod \"network-check-target-kz9ls\" (UID: \"3d82edaa-902a-4dd4-a5f3-acb327cfb733\") " pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:49.208454 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.208424 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kthkt\"" Jul 07 02:18:49.214848 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.214824 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv29t\"" Jul 07 02:18:49.215855 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.215835 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4sg5" Jul 07 02:18:49.216690 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.216669 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wz8mb" Jul 07 02:18:49.223178 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.223159 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:49.346284 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.346197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:49.548040 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.548006 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:49.742052 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:49.741942 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:50.199062 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.199034 2557 scope.go:117] "RemoveContainer" containerID="6ac645bf0ba01f2a5b6257648b4e11ca884dc4c13caaea9652a40c7c9e99e315" Jul 07 02:18:50.848200 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.848166 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:50.848200 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.848205 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:50.848447 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.848262 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:50.848447 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.848287 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:50.848447 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.848326 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:50.849041 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.849013 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714d46ff-0d58-4ab4-861d-93a62727db76-service-ca-bundle\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:50.851350 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.851317 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714d46ff-0d58-4ab4-861d-93a62727db76-metrics-certs\") pod \"router-default-7bbd949cd7-zhfd6\" (UID: \"714d46ff-0d58-4ab4-861d-93a62727db76\") " pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:50.851482 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.851456 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"image-registry-7544685bd5-x8rbm\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:50.852102 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.851994 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2841fb76-869e-4fdb-8394-38d9432dea0e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8wm6c\" (UID: \"2841fb76-869e-4fdb-8394-38d9432dea0e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:50.852102 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.852057 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f477398-c6db-427b-bca8-45d3675251e9-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-w8v9v\" (UID: \"7f477398-c6db-427b-bca8-45d3675251e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:50.936159 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.936124 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q4sg5"] Jul 07 02:18:50.948747 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.948718 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:50.951072 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:50.951043 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa97054e-2ebe-4769-9e9a-09526631f660-metrics-tls\") pod \"dns-default-c7gd2\" (UID: \"aa97054e-2ebe-4769-9e9a-09526631f660\") " pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:51.066927 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.066696 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fnv6g\"" Jul 07 02:18:51.074855 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.074829 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" Jul 07 02:18:51.109582 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.109553 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qxr5d\"" Jul 07 02:18:51.116775 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.116747 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" Jul 07 02:18:51.136713 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.136589 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-56fvm\"" Jul 07 02:18:51.141961 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.141926 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ncxp5\"" Jul 07 02:18:51.144931 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.144909 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:51.150828 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.150807 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:51.153418 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:51.153131 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2379c48f_d209_40e9_88da_f7427e4e9a18.slice/crio-0b174ebe12c458d4d9de0c2391c3b88440c6cee11e43928fd6d9f0b0f0f937b8 WatchSource:0}: Error finding container 0b174ebe12c458d4d9de0c2391c3b88440c6cee11e43928fd6d9f0b0f0f937b8: Status 404 returned error can't find the container with id 0b174ebe12c458d4d9de0c2391c3b88440c6cee11e43928fd6d9f0b0f0f937b8 Jul 07 02:18:51.156648 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.156621 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4v9ss\"" Jul 07 02:18:51.165448 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.164847 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:51.369091 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.369050 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wz8mb"] Jul 07 02:18:51.380322 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:51.380288 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06baddfd_0455_4b6d_8e02_c9fef640b572.slice/crio-6ab942418e7633cdf9d4030d70c8396d2b1bc54a3e4cd0a47c0b124a585a1066 WatchSource:0}: Error finding container 6ab942418e7633cdf9d4030d70c8396d2b1bc54a3e4cd0a47c0b124a585a1066: Status 404 returned error can't find the container with id 6ab942418e7633cdf9d4030d70c8396d2b1bc54a3e4cd0a47c0b124a585a1066 Jul 07 02:18:51.402457 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.402122 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kz9ls"] Jul 07 02:18:51.406869 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:51.406822 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d82edaa_902a_4dd4_a5f3_acb327cfb733.slice/crio-893bd1a085f43f5c0c8eadc41ecb07920cc3e44e264e72d555188c94e2abfb81 WatchSource:0}: Error finding container 893bd1a085f43f5c0c8eadc41ecb07920cc3e44e264e72d555188c94e2abfb81: Status 404 returned error can't find the container with id 893bd1a085f43f5c0c8eadc41ecb07920cc3e44e264e72d555188c94e2abfb81 Jul 07 02:18:51.439294 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.438500 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7544685bd5-x8rbm"] Jul 07 02:18:51.454027 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:51.453366 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be49692_e2f1_4968_b6e3_b5993acaeb52.slice/crio-1644672e4cd3aeaac22567fe04e7acca92b9cf04ca0e9775cdf8136c73f84d61 WatchSource:0}: Error finding container 1644672e4cd3aeaac22567fe04e7acca92b9cf04ca0e9775cdf8136c73f84d61: Status 404 returned error can't find the container with id 1644672e4cd3aeaac22567fe04e7acca92b9cf04ca0e9775cdf8136c73f84d61 Jul 07 02:18:51.460969 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.458450 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v"] Jul 07 02:18:51.463096 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.462832 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7bbd949cd7-zhfd6"] Jul 07 02:18:51.494279 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.494174 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c"] Jul 07 02:18:51.524571 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.524546 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c7gd2"] Jul 07 02:18:51.529353 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:51.529324 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa97054e_2ebe_4769_9e9a_09526631f660.slice/crio-cb3ffde2a9e4ac63de43d973e3138aa9329d302481cda47dab605cf038c8630e WatchSource:0}: Error finding container cb3ffde2a9e4ac63de43d973e3138aa9329d302481cda47dab605cf038c8630e: Status 404 returned error can't find the container with id cb3ffde2a9e4ac63de43d973e3138aa9329d302481cda47dab605cf038c8630e Jul 07 02:18:51.621121 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.621034 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh5gt" event={"ID":"3fa37f6f-940a-48bc-91bf-e04f86947ca5","Type":"ContainerStarted","Data":"1721f0730321063f896816943a2f0557fbfc6590763018c26d4699f966a3aaf7"} Jul 07 02:18:51.622450 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.622408 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" event={"ID":"7f477398-c6db-427b-bca8-45d3675251e9","Type":"ContainerStarted","Data":"2b564bc179cf5c3662dbe295e1ae208b188aa54864ebad87a0d3f7dbf69cb8cd"} Jul 07 02:18:51.624148 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.624119 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" event={"ID":"8be49692-e2f1-4968-b6e3-b5993acaeb52","Type":"ContainerStarted","Data":"6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231"} Jul 07 02:18:51.624256 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.624170 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" event={"ID":"8be49692-e2f1-4968-b6e3-b5993acaeb52","Type":"ContainerStarted","Data":"1644672e4cd3aeaac22567fe04e7acca92b9cf04ca0e9775cdf8136c73f84d61"} Jul 07 02:18:51.624256 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.624198 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:18:51.625421 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.625402 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wz8mb" event={"ID":"06baddfd-0455-4b6d-8e02-c9fef640b572","Type":"ContainerStarted","Data":"6ab942418e7633cdf9d4030d70c8396d2b1bc54a3e4cd0a47c0b124a585a1066"} Jul 07 02:18:51.627365 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.627346 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:18:51.627489 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.627458 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" event={"ID":"67e6975a-a327-4c7f-b7ce-23ab9bc3d059","Type":"ContainerStarted","Data":"51414a558ec93faa3899ce557857946fed13bb893cbfb4ce146f3288119345a7"} Jul 07 02:18:51.627755 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.627726 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:51.629299 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.629277 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" event={"ID":"714d46ff-0d58-4ab4-861d-93a62727db76","Type":"ContainerStarted","Data":"294337109a9545aa6dcc011cbbbd72b720fa65d14bc06b8cc791749c78d8228b"} Jul 07 02:18:51.629448 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.629325 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" event={"ID":"714d46ff-0d58-4ab4-861d-93a62727db76","Type":"ContainerStarted","Data":"655e268b14e254f2bd5a2103d2326ffb5a5823b68eb999aa51d9be921d75cc04"} Jul 07 02:18:51.630522 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.630493 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" event={"ID":"2841fb76-869e-4fdb-8394-38d9432dea0e","Type":"ContainerStarted","Data":"75e98cfd9f6c1f98664fcb3fce4896fa6d78566fe140326486594150b316b3e0"} Jul 07 02:18:51.632014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.631992 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kz9ls" event={"ID":"3d82edaa-902a-4dd4-a5f3-acb327cfb733","Type":"ContainerStarted","Data":"6d92cf90255b90c845589ba4f6ae1d4a18e8cb75a64be901a611264befb4e92a"} Jul 07 02:18:51.632112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.632032 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kz9ls" event={"ID":"3d82edaa-902a-4dd4-a5f3-acb327cfb733","Type":"ContainerStarted","Data":"893bd1a085f43f5c0c8eadc41ecb07920cc3e44e264e72d555188c94e2abfb81"} Jul 07 02:18:51.632112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.632057 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:18:51.633204 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.633181 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c7gd2" event={"ID":"aa97054e-2ebe-4769-9e9a-09526631f660","Type":"ContainerStarted","Data":"cb3ffde2a9e4ac63de43d973e3138aa9329d302481cda47dab605cf038c8630e"} Jul 07 02:18:51.634309 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.634290 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4sg5" event={"ID":"2379c48f-d209-40e9-88da-f7427e4e9a18","Type":"ContainerStarted","Data":"0b174ebe12c458d4d9de0c2391c3b88440c6cee11e43928fd6d9f0b0f0f937b8"} Jul 07 02:18:51.650431 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.647691 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" podStartSLOduration=44.647674154 podStartE2EDuration="44.647674154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:51.646132642 +0000 UTC m=+68.009335291" watchObservedRunningTime="2026-07-07 02:18:51.647674154 +0000 UTC m=+68.010876739" Jul 07 02:18:51.667779 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.667730 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" podStartSLOduration=32.853542364 podStartE2EDuration="42.667714422s" totalImagesPullingTime="9.814172058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:51.665822662 +0000 UTC m=+68.029025248" watchObservedRunningTime="2026-07-07 02:18:51.667714422 +0000 UTC m=+68.030917006" Jul 07 02:18:51.684651 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.684591 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" podStartSLOduration=42.684573057 podStartE2EDuration="42.684573057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:51.682540589 +0000 UTC m=+68.045743174" watchObservedRunningTime="2026-07-07 02:18:51.684573057 +0000 UTC m=+68.047775648" Jul 07 02:18:51.697161 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:51.697113 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kz9ls" podStartSLOduration=67.697098526 podStartE2EDuration="1m7.697098526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:51.696418419 +0000 UTC m=+68.059621003" watchObservedRunningTime="2026-07-07 02:18:51.697098526 +0000 UTC m=+68.060301127" Jul 07 02:18:52.151925 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.151890 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:52.154762 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.154733 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:52.203605 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.203577 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-kxtqh" Jul 07 02:18:52.383917 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.383833 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:52.483857 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.483762 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:52.542706 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.542648 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:52.644532 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.644449 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh5gt" event={"ID":"3fa37f6f-940a-48bc-91bf-e04f86947ca5","Type":"ContainerStarted","Data":"b36fb4bdc7b7d36d8ea51322d62be2a5ca5bde52f4b37479caabda46daa87163"} Jul 07 02:18:52.644974 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.644915 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:52.646444 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.646425 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7bbd949cd7-zhfd6" Jul 07 02:18:52.673708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.673669 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:52.915435 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:52.915366 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:53.017499 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.017433 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:53.118673 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.118596 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:53.258315 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.258282 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:53.475083 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.474975 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:53.536429 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.536252 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:53.654730 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.654619 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4sg5" event={"ID":"2379c48f-d209-40e9-88da-f7427e4e9a18","Type":"ContainerStarted","Data":"f8126b816cc81da31d178fcc6d73c0d1d644fd7512fac7797379eb6c74d67ca8"} Jul 07 02:18:53.654730 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.654671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4sg5" event={"ID":"2379c48f-d209-40e9-88da-f7427e4e9a18","Type":"ContainerStarted","Data":"04bd4af896b6571045a253768ea4fd9c365bd04078c5d774691b0b84b0c99934"} Jul 07 02:18:53.671867 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.671807 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q4sg5" podStartSLOduration=68.052023986 podStartE2EDuration="1m9.671785204s" totalImagesPullingTime="1.619761218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:53.671347069 +0000 UTC m=+70.034549655" watchObservedRunningTime="2026-07-07 02:18:53.671785204 +0000 UTC m=+70.034987790" Jul 07 02:18:53.860170 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:53.860057 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:54.058177 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:54.058141 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:54.112672 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:54.112583 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:54.285879 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:54.285830 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:54.339025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:54.338984 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:57.883412 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:57.883347 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:57.982594 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:57.982563 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:58.081930 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.081900 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:58.184600 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.184569 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:58.291030 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.290965 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:58.573566 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.573530 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:58.674576 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.674481 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:58.675629 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.675600 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" event={"ID":"2841fb76-869e-4fdb-8394-38d9432dea0e","Type":"ContainerStarted","Data":"bfa6d878373b4201f1f5415fb7a7403dfa3a48b9691a78b5654c605c57f24183"} Jul 07 02:18:58.677453 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.677424 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c7gd2" event={"ID":"aa97054e-2ebe-4769-9e9a-09526631f660","Type":"ContainerStarted","Data":"b0e7d98441c5ece463dbc6b0efe3adf2aaaa04d7506d45a1506259ede533c737"} Jul 07 02:18:58.677613 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.677476 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c7gd2" event={"ID":"aa97054e-2ebe-4769-9e9a-09526631f660","Type":"ContainerStarted","Data":"c6cfb9f58382cd131249e5ea059dff54cf4494da80fc58a722db995cdc905e27"} Jul 07 02:18:58.677613 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.677505 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c7gd2" Jul 07 02:18:58.679559 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.679536 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh5gt" event={"ID":"3fa37f6f-940a-48bc-91bf-e04f86947ca5","Type":"ContainerStarted","Data":"a87861e6c5a80d21d434e5d24c1a720a2fbaa2bcbaa6461c02dd78f0147e71e3"} Jul 07 02:18:58.681211 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.681189 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" event={"ID":"7f477398-c6db-427b-bca8-45d3675251e9","Type":"ContainerStarted","Data":"336213e7c9d820b5bc3ae0bbbd18f6dbb88760fa9b484a29634618b63bdecc4f"} Jul 07 02:18:58.681309 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.681235 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" event={"ID":"7f477398-c6db-427b-bca8-45d3675251e9","Type":"ContainerStarted","Data":"4320460389b79cd0bcd6fd21e9067197a621c2f3e79f0f0985fc8f11948d855a"} Jul 07 02:18:58.682718 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.682696 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wz8mb" event={"ID":"06baddfd-0455-4b6d-8e02-c9fef640b572","Type":"ContainerStarted","Data":"6e039ab12a6995c16ad16ad4fb32f9d17c915129dc7ea3f46f536b6b6aabb1ee"} Jul 07 02:18:58.711001 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.710957 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8wm6c" podStartSLOduration=43.121719569 podStartE2EDuration="49.710943375s" totalImagesPullingTime="6.589223806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:58.710621435 +0000 UTC m=+75.073824019" watchObservedRunningTime="2026-07-07 02:18:58.710943375 +0000 UTC m=+75.074145959" Jul 07 02:18:58.754531 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.754477 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-w8v9v" podStartSLOduration=43.173792891 podStartE2EDuration="49.754456428s" totalImagesPullingTime="6.580663537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:58.753483278 +0000 UTC m=+75.116685864" watchObservedRunningTime="2026-07-07 02:18:58.754456428 +0000 UTC m=+75.117659013" Jul 07 02:18:58.830749 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.830720 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:58.838394 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.838322 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sh5gt" podStartSLOduration=17.263742376 podStartE2EDuration="27.838288732s" totalImagesPullingTime="10.574546356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:58.829362766 +0000 UTC m=+75.192565351" watchObservedRunningTime="2026-07-07 02:18:58.838288732 +0000 UTC m=+75.201491317" Jul 07 02:18:58.839470 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.839452 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d"] Jul 07 02:18:58.850507 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.850489 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" Jul 07 02:18:58.861012 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.860991 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 07 02:18:58.861290 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.861275 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-f92r9\"" Jul 07 02:18:58.884531 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.884480 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wz8mb" podStartSLOduration=67.1642508 podStartE2EDuration="1m13.884451347s" totalImagesPullingTime="6.720200547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:58.878173182 +0000 UTC m=+75.241375765" watchObservedRunningTime="2026-07-07 02:18:58.884451347 +0000 UTC m=+75.247653934" Jul 07 02:18:58.886655 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.886627 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d"] Jul 07 02:18:58.923451 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.923426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a8302b01-d8c8-42c8-8c93-d3aaad238b2f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cpk2d\" (UID: \"a8302b01-d8c8-42c8-8c93-d3aaad238b2f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" Jul 07 02:18:58.929095 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.928993 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c7gd2" podStartSLOduration=34.367080565 podStartE2EDuration="40.92897392s" totalImagesPullingTime="6.561893355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:58.927195166 +0000 UTC m=+75.290397746" watchObservedRunningTime="2026-07-07 02:18:58.92897392 +0000 UTC m=+75.292176515" Jul 07 02:18:58.983800 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:58.983760 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:59.023906 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:59.023877 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a8302b01-d8c8-42c8-8c93-d3aaad238b2f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cpk2d\" (UID: \"a8302b01-d8c8-42c8-8c93-d3aaad238b2f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" Jul 07 02:18:59.026327 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:59.026306 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a8302b01-d8c8-42c8-8c93-d3aaad238b2f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cpk2d\" (UID: \"a8302b01-d8c8-42c8-8c93-d3aaad238b2f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" Jul 07 02:18:59.160186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:59.160159 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" Jul 07 02:18:59.303062 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:59.303025 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d"] Jul 07 02:18:59.307073 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:18:59.307046 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8302b01_d8c8_42c8_8c93_d3aaad238b2f.slice/crio-61c8111fb8c7bc1439bd74c0c58413b70972d4ab6b5cdfbf1f34cc0f683de3b9 WatchSource:0}: Error finding container 61c8111fb8c7bc1439bd74c0c58413b70972d4ab6b5cdfbf1f34cc0f683de3b9: Status 404 returned error can't find the container with id 61c8111fb8c7bc1439bd74c0c58413b70972d4ab6b5cdfbf1f34cc0f683de3b9 Jul 07 02:18:59.687681 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:18:59.687641 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" event={"ID":"a8302b01-d8c8-42c8-8c93-d3aaad238b2f","Type":"ContainerStarted","Data":"61c8111fb8c7bc1439bd74c0c58413b70972d4ab6b5cdfbf1f34cc0f683de3b9"} Jul 07 02:19:01.696765 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:01.696734 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" event={"ID":"a8302b01-d8c8-42c8-8c93-d3aaad238b2f","Type":"ContainerStarted","Data":"bb992a85d83a37f3c4c00e6c5aa098762fe1abd9d126e28624b04b8e08097d6c"} Jul 07 02:19:01.697184 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:01.696887 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" Jul 07 02:19:01.701648 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:01.701627 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" Jul 07 02:19:01.738573 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:01.738524 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cpk2d" podStartSLOduration=2.045177833 podStartE2EDuration="3.738506597s" totalImagesPullingTime="1.693328764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:01.73722478 +0000 UTC m=+78.100427364" watchObservedRunningTime="2026-07-07 02:19:01.738506597 +0000 UTC m=+78.101709181" Jul 07 02:19:01.979968 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:01.979877 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:02.080523 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.080485 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:02.816289 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.816249 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:02.838980 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.838944 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t"] Jul 07 02:19:02.858055 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.858029 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t"] Jul 07 02:19:02.858194 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.858143 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:02.862014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.861988 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 02:19:02.862158 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.861988 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 02:19:02.862158 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.862090 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2r9wj\"" Jul 07 02:19:02.862158 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.861996 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 02:19:02.881547 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.881520 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:02.955237 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.955203 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nnmp\" (UniqueName: \"kubernetes.io/projected/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-kube-api-access-2nnmp\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:02.955416 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.955271 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:02.955416 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.955301 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:02.955416 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:02.955321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.056300 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.056263 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.056300 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.056304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.056510 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.056325 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.056510 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.056354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nnmp\" (UniqueName: \"kubernetes.io/projected/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-kube-api-access-2nnmp\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.056510 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:19:03.056486 2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 07 02:19:03.056602 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:19:03.056554 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-tls podName:9c2ea828-653a-4ff0-b28c-a3ce74edcbee nodeName:}" failed. No retries permitted until 2026-07-07 02:19:03.556535553 +0000 UTC m=+79.919738128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-frg8t" (UID: "9c2ea828-653a-4ff0-b28c-a3ce74edcbee") : secret "prometheus-operator-tls" not found Jul 07 02:19:03.056974 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.056942 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.058741 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.058722 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.066059 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.066034 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nnmp\" (UniqueName: \"kubernetes.io/projected/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-kube-api-access-2nnmp\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.146071 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.145991 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:03.200636 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.200600 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:03.256457 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.256417 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:03.559933 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.559894 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.562293 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.562260 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c2ea828-653a-4ff0-b28c-a3ce74edcbee-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-frg8t\" (UID: \"9c2ea828-653a-4ff0-b28c-a3ce74edcbee\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.767354 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.767318 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" Jul 07 02:19:03.923961 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:03.923928 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t"] Jul 07 02:19:03.926738 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:19:03.926704 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2ea828_653a_4ff0_b28c_a3ce74edcbee.slice/crio-05659b06faf93181f907a44c18fcc4273bb40bfe3bd03a815a10e5e1c454c846 WatchSource:0}: Error finding container 05659b06faf93181f907a44c18fcc4273bb40bfe3bd03a815a10e5e1c454c846: Status 404 returned error can't find the container with id 05659b06faf93181f907a44c18fcc4273bb40bfe3bd03a815a10e5e1c454c846 Jul 07 02:19:04.710803 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:04.710761 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" event={"ID":"9c2ea828-653a-4ff0-b28c-a3ce74edcbee","Type":"ContainerStarted","Data":"05659b06faf93181f907a44c18fcc4273bb40bfe3bd03a815a10e5e1c454c846"} Jul 07 02:19:04.714616 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:04.714585 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:04.774960 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:04.774911 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:04.881009 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:04.880961 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:04.948678 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:04.948634 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:05.969680 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:05.969599 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:06.721715 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:06.721680 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" event={"ID":"9c2ea828-653a-4ff0-b28c-a3ce74edcbee","Type":"ContainerStarted","Data":"ad2cf2f0bbc9be661a0d987de58449a9a24d8585cb1ad12c7208aafc137547d9"} Jul 07 02:19:06.721877 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:06.721734 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" event={"ID":"9c2ea828-653a-4ff0-b28c-a3ce74edcbee","Type":"ContainerStarted","Data":"0de71fb5f4dc2b846107224f06a4b8f801259e4a9003674bfdea9b5bdd668385"} Jul 07 02:19:06.737365 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:06.737309 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-frg8t" podStartSLOduration=3.007775313 podStartE2EDuration="4.73729176s" totalImagesPullingTime="1.729516447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:06.737281497 +0000 UTC m=+83.100484081" watchObservedRunningTime="2026-07-07 02:19:06.73729176 +0000 UTC m=+83.100494344" Jul 07 02:19:07.019546 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:07.019509 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:07.081894 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:07.081849 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:07.950311 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:07.950280 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.052340 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.052302 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.153436 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.153397 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.250983 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.250952 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rskxb"] Jul 07 02:19:08.254241 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.254220 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.257215 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.257187 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8twrq\"" Jul 07 02:19:08.257328 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.257188 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 02:19:08.257545 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.257527 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 02:19:08.257700 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.257685 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 02:19:08.282151 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.282118 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.373742 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.373709 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.399728 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ca8b31-bf65-400c-a1d0-2613491df2b1-metrics-client-ca\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.399862 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399740 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-sys\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.399862 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399777 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.399862 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399829 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-textfile\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.399862 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-wtmp\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.400021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399865 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.400021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-root\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.400021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399924 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.400021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.399945 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqrs\" (UniqueName: \"kubernetes.io/projected/32ca8b31-bf65-400c-a1d0-2613491df2b1-kube-api-access-mnqrs\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.473661 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.473627 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.500700 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500669 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.500804 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500707 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-textfile\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.500804 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500728 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-wtmp\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.500804 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.500804 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-root\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500820 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500845 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqrs\" (UniqueName: \"kubernetes.io/projected/32ca8b31-bf65-400c-a1d0-2613491df2b1-kube-api-access-mnqrs\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500851 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-wtmp\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500888 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ca8b31-bf65-400c-a1d0-2613491df2b1-metrics-client-ca\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500910 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-root\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:19:08.500916 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500928 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-sys\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:19:08.500984 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls podName:32ca8b31-bf65-400c-a1d0-2613491df2b1 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:09.000963998 +0000 UTC m=+85.364166562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls") pod "node-exporter-rskxb" (UID: "32ca8b31-bf65-400c-a1d0-2613491df2b1") : secret "node-exporter-tls" not found Jul 07 02:19:08.501025 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.500998 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32ca8b31-bf65-400c-a1d0-2613491df2b1-sys\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501339 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.501048 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-textfile\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501447 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.501430 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32ca8b31-bf65-400c-a1d0-2613491df2b1-metrics-client-ca\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.501505 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.501487 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.503124 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.503102 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.509077 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.509055 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqrs\" (UniqueName: \"kubernetes.io/projected/32ca8b31-bf65-400c-a1d0-2613491df2b1-kube-api-access-mnqrs\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:08.580965 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.580930 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.690169 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.690138 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c7gd2" Jul 07 02:19:08.877556 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:08.877464 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:09.005911 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:09.005873 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:09.006083 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:19:09.006031 2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 02:19:09.006136 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:19:09.006108 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls podName:32ca8b31-bf65-400c-a1d0-2613491df2b1 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:10.006090649 +0000 UTC m=+86.369293211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls") pod "node-exporter-rskxb" (UID: "32ca8b31-bf65-400c-a1d0-2613491df2b1") : secret "node-exporter-tls" not found Jul 07 02:19:09.122204 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:09.122169 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:09.222130 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:09.222037 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:09.284733 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:09.284698 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:09.457067 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:09.457038 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:09.688859 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:09.688826 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:09.789438 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:09.789403 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:10.013927 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.013889 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:10.016342 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.016319 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32ca8b31-bf65-400c-a1d0-2613491df2b1-node-exporter-tls\") pod \"node-exporter-rskxb\" (UID: \"32ca8b31-bf65-400c-a1d0-2613491df2b1\") " pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:10.063582 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.063545 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rskxb" Jul 07 02:19:10.075073 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:19:10.075045 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ca8b31_bf65_400c_a1d0_2613491df2b1.slice/crio-e06cfc15ac898c9d8531cbc2152a7da015974c8cad467f190fbe3123e0ce4895 WatchSource:0}: Error finding container e06cfc15ac898c9d8531cbc2152a7da015974c8cad467f190fbe3123e0ce4895: Status 404 returned error can't find the container with id e06cfc15ac898c9d8531cbc2152a7da015974c8cad467f190fbe3123e0ce4895 Jul 07 02:19:10.135533 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.135496 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:10.201584 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.201552 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:10.735998 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.735958 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:10.737971 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.737919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rskxb" event={"ID":"32ca8b31-bf65-400c-a1d0-2613491df2b1","Type":"ContainerStarted","Data":"e06cfc15ac898c9d8531cbc2152a7da015974c8cad467f190fbe3123e0ce4895"} Jul 07 02:19:10.831503 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:10.831449 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:11.132667 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:11.132637 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:11.214739 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:11.214710 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:11.329550 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:11.329513 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:11.443777 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:11.443688 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:11.743566 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:11.743469 2557 generic.go:356] "Generic (PLEG): container finished" podID="32ca8b31-bf65-400c-a1d0-2613491df2b1" containerID="840fc2d6cfe2218e5abc3ac18c7ff84639c6007ae360487bb5d4e7290fd5a611" exitCode=0 Jul 07 02:19:11.743717 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:11.743566 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rskxb" event={"ID":"32ca8b31-bf65-400c-a1d0-2613491df2b1","Type":"ContainerDied","Data":"840fc2d6cfe2218e5abc3ac18c7ff84639c6007ae360487bb5d4e7290fd5a611"} Jul 07 02:19:12.649391 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:12.649327 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:19:12.749445 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:12.749413 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rskxb" event={"ID":"32ca8b31-bf65-400c-a1d0-2613491df2b1","Type":"ContainerStarted","Data":"8edb477e2815507fd10b6fb1a57cb00d90f8c58bafa485abd50cb1b43dda6273"} Jul 07 02:19:12.749628 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:12.749465 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rskxb" event={"ID":"32ca8b31-bf65-400c-a1d0-2613491df2b1","Type":"ContainerStarted","Data":"54dd6946d51f3fd480e99ec17f4f2aa6185467252598127469d2dad2d0ac0365"} Jul 07 02:19:12.768680 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:12.768633 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rskxb" podStartSLOduration=3.645746235 podStartE2EDuration="4.768617265s" totalImagesPullingTime="1.055321888s" totalInitContainerRuntime="67.549142ms" isStatefulPod=true podCreationTimestamp="2026-07-07 02:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:12.766658866 +0000 UTC m=+89.129861451" watchObservedRunningTime="2026-07-07 02:19:12.768617265 +0000 UTC m=+89.131819853" Jul 07 02:19:12.814768 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:12.814734 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:12.910149 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:12.910060 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:12.991155 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:12.991117 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:13.124292 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:13.124253 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:13.181060 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:13.180980 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:14.422725 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:14.422688 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:14.769466 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:14.769429 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:14.870923 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:14.870889 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:17.807946 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:17.807915 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:17.862429 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:17.862394 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:18.703699 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:18.700135 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:18.760150 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:18.760110 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:19.884726 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:19.884694 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7544685bd5-x8rbm"] Jul 07 02:19:19.961487 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:19.961449 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:20.057389 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:20.057330 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:21.628960 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:21.628922 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:21.729206 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:21.729165 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:22.141338 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:22.141288 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:22.242890 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:22.242858 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:22.647977 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:22.647948 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kz9ls" Jul 07 02:19:22.919306 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:22.919236 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:22.977243 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:22.977205 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:25.023172 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:25.023141 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:25.114466 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:25.114435 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:28.852359 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:28.852324 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:28.909584 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:28.909545 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:30.737148 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:30.737083 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:30.803645 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:30.803606 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:37.236127 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:37.236087 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:37.336648 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:37.336607 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:37.395147 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:37.395121 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:44.905392 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:44.905313 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" podUID="8be49692-e2f1-4968-b6e3-b5993acaeb52" containerName="registry" containerID="cri-o://6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231" gracePeriod=30 Jul 07 02:19:45.156250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.156190 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:19:45.180919 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.180882 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8be49692-e2f1-4968-b6e3-b5993acaeb52-ca-trust-extracted\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.181084 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.180944 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-trusted-ca\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.181084 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.180971 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.181084 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.181068 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-certificates\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.181243 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.181117 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9pz2\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-kube-api-access-b9pz2\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.181243 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.181173 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-installation-pull-secrets\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.181243 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.181216 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-bound-sa-token\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.181424 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.181266 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-image-registry-private-configuration\") pod \"8be49692-e2f1-4968-b6e3-b5993acaeb52\" (UID: \"8be49692-e2f1-4968-b6e3-b5993acaeb52\") " Jul 07 02:19:45.182099 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.181693 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:19:45.182099 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.182063 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:19:45.185046 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.184933 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:19:45.185550 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.185183 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:19:45.186094 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.186022 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:19:45.186094 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.186043 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-kube-api-access-b9pz2" (OuterVolumeSpecName: "kube-api-access-b9pz2") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "kube-api-access-b9pz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:19:45.186472 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.186445 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:19:45.191442 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.191413 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be49692-e2f1-4968-b6e3-b5993acaeb52-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8be49692-e2f1-4968-b6e3-b5993acaeb52" (UID: "8be49692-e2f1-4968-b6e3-b5993acaeb52"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:19:45.282212 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282176 2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-installation-pull-secrets\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.282212 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282206 2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-bound-sa-token\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.282212 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282217 2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8be49692-e2f1-4968-b6e3-b5993acaeb52-image-registry-private-configuration\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.282460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282230 2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8be49692-e2f1-4968-b6e3-b5993acaeb52-ca-trust-extracted\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.282460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282239 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-trusted-ca\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.282460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282246 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.282460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282254 2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8be49692-e2f1-4968-b6e3-b5993acaeb52-registry-certificates\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.282460 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.282262 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9pz2\" (UniqueName: \"kubernetes.io/projected/8be49692-e2f1-4968-b6e3-b5993acaeb52-kube-api-access-b9pz2\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:19:45.871274 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.871233 2557 generic.go:356] "Generic (PLEG): container finished" podID="8be49692-e2f1-4968-b6e3-b5993acaeb52" containerID="6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231" exitCode=0 Jul 07 02:19:45.871472 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.871304 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" Jul 07 02:19:45.871472 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.871316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" event={"ID":"8be49692-e2f1-4968-b6e3-b5993acaeb52","Type":"ContainerDied","Data":"6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231"} Jul 07 02:19:45.871472 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.871424 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7544685bd5-x8rbm" event={"ID":"8be49692-e2f1-4968-b6e3-b5993acaeb52","Type":"ContainerDied","Data":"1644672e4cd3aeaac22567fe04e7acca92b9cf04ca0e9775cdf8136c73f84d61"} Jul 07 02:19:45.871472 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.871460 2557 scope.go:117] "RemoveContainer" containerID="6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231" Jul 07 02:19:45.880334 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.880317 2557 scope.go:117] "RemoveContainer" containerID="6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231" Jul 07 02:19:45.880611 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:19:45.880584 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231\": container with ID starting with 6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231 not found: ID does not exist" containerID="6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231" Jul 07 02:19:45.880662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.880622 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231"} err="failed to get container status \"6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231\": rpc error: code = NotFound desc = could not find container \"6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231\": container with ID starting with 6f2c683a2fbcf1067b9a3b7ad0b48af69c72c7cf84a7e9f6a376a76fd94c4231 not found: ID does not exist" Jul 07 02:19:45.893502 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.893480 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7544685bd5-x8rbm"] Jul 07 02:19:45.903912 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:45.903889 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7544685bd5-x8rbm"] Jul 07 02:19:46.198959 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:46.198868 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be49692-e2f1-4968-b6e3-b5993acaeb52" path="/var/lib/kubelet/pods/8be49692-e2f1-4968-b6e3-b5993acaeb52/volumes" Jul 07 02:19:48.969407 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:48.969334 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:50.891095 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:50.891066 2557 generic.go:356] "Generic (PLEG): container finished" podID="ba3f3d85-b305-4bb7-a031-640fc9f4f2e3" containerID="480bf0271ee8f6884074e291895fe537f1cb0e4f1200a7903b63454675fd36ec" exitCode=0 Jul 07 02:19:50.891490 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:50.891109 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" event={"ID":"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3","Type":"ContainerDied","Data":"480bf0271ee8f6884074e291895fe537f1cb0e4f1200a7903b63454675fd36ec"} Jul 07 02:19:50.891490 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:50.891448 2557 scope.go:117] "RemoveContainer" containerID="480bf0271ee8f6884074e291895fe537f1cb0e4f1200a7903b63454675fd36ec" Jul 07 02:19:51.590260 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:51.590226 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bbd949cd7-zhfd6_714d46ff-0d58-4ab4-861d-93a62727db76/router/0.log" Jul 07 02:19:51.596059 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:51.596035 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8nsjg_2b5d5655-53e3-4359-adde-5e1da624dd44/serve-healthcheck-canary/0.log" Jul 07 02:19:51.896477 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:51.896401 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-94vfd" event={"ID":"ba3f3d85-b305-4bb7-a031-640fc9f4f2e3","Type":"ContainerStarted","Data":"c7eef4442c0ee3e637f052c336a375c34410a959bb773e31dfa1fca263389e77"} Jul 07 02:19:52.919630 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:52.919599 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:52.979913 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:19:52.979874 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:00.931031 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:00.930999 2557 generic.go:356] "Generic (PLEG): container finished" podID="647ff255-7388-4f76-8076-fefe3bc177b8" containerID="5500f66a80a74596126ae8257a22b4b1769ce4797aeb07fa606d4bff7863fe3c" exitCode=0 Jul 07 02:20:00.931522 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:00.931079 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" event={"ID":"647ff255-7388-4f76-8076-fefe3bc177b8","Type":"ContainerDied","Data":"5500f66a80a74596126ae8257a22b4b1769ce4797aeb07fa606d4bff7863fe3c"} Jul 07 02:20:00.931522 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:00.931439 2557 scope.go:117] "RemoveContainer" containerID="5500f66a80a74596126ae8257a22b4b1769ce4797aeb07fa606d4bff7863fe3c" Jul 07 02:20:00.932620 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:00.932595 2557 generic.go:356] "Generic (PLEG): container finished" podID="9660f59c-3137-4943-bcc2-a94a66b279fd" containerID="27a02d5959329e1889f2807d1fde61a2cb4d71dca0e967a35c35adbcdad3b803" exitCode=0 Jul 07 02:20:00.932711 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:00.932663 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" event={"ID":"9660f59c-3137-4943-bcc2-a94a66b279fd","Type":"ContainerDied","Data":"27a02d5959329e1889f2807d1fde61a2cb4d71dca0e967a35c35adbcdad3b803"} Jul 07 02:20:00.932961 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:00.932946 2557 scope.go:117] "RemoveContainer" containerID="27a02d5959329e1889f2807d1fde61a2cb4d71dca0e967a35c35adbcdad3b803" Jul 07 02:20:01.938549 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:01.938512 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-7l7pw" event={"ID":"647ff255-7388-4f76-8076-fefe3bc177b8","Type":"ContainerStarted","Data":"3143da79b93d98beb1897a9c93125481ef124b0863c89192d7add0ee7826d78e"} Jul 07 02:20:01.940316 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:01.940293 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xc9b5" event={"ID":"9660f59c-3137-4943-bcc2-a94a66b279fd","Type":"ContainerStarted","Data":"0f0b9ebdc19583c88d2ca5ceb2a9cc0d7c70c61db0d7fd10d5712f7507c9b9e9"} Jul 07 02:20:02.719388 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:02.719340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:02.818923 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:02.818883 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:02.883080 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:02.883046 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:24.310436 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:24.310403 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:24.379726 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:24.379692 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:36.253975 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:36.253892 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:36.322620 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:36.322587 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:52.700146 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:52.700108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:52.800728 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:52.800685 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:52.902197 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:52.902162 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:53.001866 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:53.001827 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:53.065762 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:53.065730 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:53.401835 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:53.401744 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:53.454997 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:53.454963 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:55.022384 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:55.022336 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:55.112855 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:20:55.112824 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:03.486112 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:03.486065 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:03.586255 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:03.586214 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:21.946346 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:21.946312 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:21.997106 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:21.997069 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:32.173980 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:32.173949 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:32.259320 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:32.259283 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:36.393941 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.393896 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:36.476105 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.476074 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:36.479573 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.479547 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9"] Jul 07 02:21:36.479863 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.479850 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be49692-e2f1-4968-b6e3-b5993acaeb52" containerName="registry" Jul 07 02:21:36.479902 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.479865 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be49692-e2f1-4968-b6e3-b5993acaeb52" containerName="registry" Jul 07 02:21:36.479940 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.479917 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be49692-e2f1-4968-b6e3-b5993acaeb52" containerName="registry" Jul 07 02:21:36.482497 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.482483 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.485298 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.485274 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 02:21:36.486562 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.486543 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 02:21:36.486632 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.486543 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 02:21:36.486702 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.486544 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 02:21:36.496450 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.496423 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9"] Jul 07 02:21:36.497750 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.497716 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r"] Jul 07 02:21:36.500585 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.500566 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.503507 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.503482 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 02:21:36.503507 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.503503 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 02:21:36.503712 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.503607 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 02:21:36.503810 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.503739 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 02:21:36.515997 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.515960 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r"] Jul 07 02:21:36.541361 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.541331 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:21:36.641447 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641410 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtndw\" (UniqueName: \"kubernetes.io/projected/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-kube-api-access-jtndw\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.641447 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641450 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bfb8715-470c-450d-a274-b1080f2b4414-tmp\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.641662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641468 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvxl\" (UniqueName: \"kubernetes.io/projected/7bfb8715-470c-450d-a274-b1080f2b4414-kube-api-access-xqvxl\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.641662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-additional-service-ca\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.641662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641515 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-ca\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.641662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641534 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-hub\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.641662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641557 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.641662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641620 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.641843 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.641670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7bfb8715-470c-450d-a274-b1080f2b4414-klusterlet-config\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.742793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.742704 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bfb8715-470c-450d-a274-b1080f2b4414-tmp\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.742793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.742751 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvxl\" (UniqueName: \"kubernetes.io/projected/7bfb8715-470c-450d-a274-b1080f2b4414-kube-api-access-xqvxl\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.742793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.742774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-additional-service-ca\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.743007 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.742904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-ca\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.743007 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.742959 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-hub\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.743110 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.743006 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.743110 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.743042 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.743211 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.743182 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bfb8715-470c-450d-a274-b1080f2b4414-tmp\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.743263 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.743181 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7bfb8715-470c-450d-a274-b1080f2b4414-klusterlet-config\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.743320 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.743274 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtndw\" (UniqueName: \"kubernetes.io/projected/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-kube-api-access-jtndw\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.743499 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.743473 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-additional-service-ca\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.745681 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.745659 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-ca\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.745996 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.745973 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.746067 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.746034 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-hub\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.746620 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.746599 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7bfb8715-470c-450d-a274-b1080f2b4414-klusterlet-config\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.751605 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.751580 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.753523 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.753488 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvxl\" (UniqueName: \"kubernetes.io/projected/7bfb8715-470c-450d-a274-b1080f2b4414-kube-api-access-xqvxl\") pod \"klusterlet-addon-workmgr-565f4588f6-7pgz9\" (UID: \"7bfb8715-470c-450d-a274-b1080f2b4414\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.753621 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.753599 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtndw\" (UniqueName: \"kubernetes.io/projected/f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075-kube-api-access-jtndw\") pod \"cluster-proxy-proxy-agent-98d596ddb-5556r\" (UID: \"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.792028 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.792000 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:36.822391 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.822349 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" Jul 07 02:21:36.928095 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.928059 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9"] Jul 07 02:21:36.931500 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:21:36.931469 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfb8715_470c_450d_a274_b1080f2b4414.slice/crio-e9621f2e22632ee4bb58eadee31bf8db771d4f60a9e2a496404dea1d262c5ce7 WatchSource:0}: Error finding container e9621f2e22632ee4bb58eadee31bf8db771d4f60a9e2a496404dea1d262c5ce7: Status 404 returned error can't find the container with id e9621f2e22632ee4bb58eadee31bf8db771d4f60a9e2a496404dea1d262c5ce7 Jul 07 02:21:36.963720 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:36.963695 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r"] Jul 07 02:21:36.966206 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:21:36.966179 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5bf340a_4d98_4ec1_9fa9_44a8d3e7c075.slice/crio-68a109b2c972109a1a704f154e3730cacce60064c6b1828d8dc50ae8e8added8 WatchSource:0}: Error finding container 68a109b2c972109a1a704f154e3730cacce60064c6b1828d8dc50ae8e8added8: Status 404 returned error can't find the container with id 68a109b2c972109a1a704f154e3730cacce60064c6b1828d8dc50ae8e8added8 Jul 07 02:21:37.294503 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:37.294470 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" event={"ID":"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075","Type":"ContainerStarted","Data":"68a109b2c972109a1a704f154e3730cacce60064c6b1828d8dc50ae8e8added8"} Jul 07 02:21:37.295535 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:37.295512 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" event={"ID":"7bfb8715-470c-450d-a274-b1080f2b4414","Type":"ContainerStarted","Data":"e9621f2e22632ee4bb58eadee31bf8db771d4f60a9e2a496404dea1d262c5ce7"} Jul 07 02:21:41.315530 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:41.315486 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" event={"ID":"7bfb8715-470c-450d-a274-b1080f2b4414","Type":"ContainerStarted","Data":"b6bcac92b1a758219a1ef84ac805bb406843b24ecd9456b0cb8ce2fc42d0cbc3"} Jul 07 02:21:41.316015 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:41.315738 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:41.318106 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:41.318081 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" Jul 07 02:21:41.333208 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:41.333140 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565f4588f6-7pgz9" podStartSLOduration=1.672380792 podStartE2EDuration="5.333120171s" totalImagesPullingTime="3.660739379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:21:41.332656177 +0000 UTC m=+237.695858762" watchObservedRunningTime="2026-07-07 02:21:41.333120171 +0000 UTC m=+237.696322757" Jul 07 02:21:42.321645 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:42.321548 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" event={"ID":"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075","Type":"ContainerStarted","Data":"056e7a72d30990423f216f7ec5bb7f2b2559615992cf9abaac426a93e5d6ed49"} Jul 07 02:21:42.321645 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:42.321611 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" event={"ID":"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075","Type":"ContainerStarted","Data":"205d35d00e7c72ef8afbc715285898cf0d46e4f36650833169b42a66a34b7553"} Jul 07 02:21:42.321645 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:42.321625 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" event={"ID":"f5bf340a-4d98-4ec1-9fa9-44a8d3e7c075","Type":"ContainerStarted","Data":"c734001ec977f24162c869f62b1c7a54b99d7b1ccba158b7d334f435aa8fd245"} Jul 07 02:21:42.341676 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:21:42.341631 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98d596ddb-5556r" podStartSLOduration=1.278227608 podStartE2EDuration="6.341616479s" totalImagesPullingTime="5.063388871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:21:42.340214993 +0000 UTC m=+238.703417578" watchObservedRunningTime="2026-07-07 02:21:42.341616479 +0000 UTC m=+238.704819085" Jul 07 02:22:20.421629 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:20.421544 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:20.478248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:20.478214 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:20.897588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:20.897557 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:20.999651 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:20.999612 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:21.099191 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:21.099152 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:30.388227 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:30.388181 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:30.489045 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:30.489012 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:44.067085 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:44.067054 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:22:44.074196 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:44.074173 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:22:44.078027 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:44.077995 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 02:22:48.058032 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:48.057969 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:49.302983 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:49.302955 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:49.661532 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:49.661441 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:52.632772 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:52.632731 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:52.903087 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:52.902993 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.098987 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:53.098950 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.199767 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:53.199672 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.476207 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:53.476118 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.572271 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:53.572236 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.813570 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:53.813539 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.910615 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:53.910581 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:54.010397 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:54.010342 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:54.109878 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:54.109788 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:54.209612 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:22:54.209575 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:00.993284 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:00.993220 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:01.125521 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:01.125482 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:01.216954 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:01.216907 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:01.638709 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:01.638671 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:01.737631 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:01.737584 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:27.358962 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:27.358904 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:27.429726 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:27.429686 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:32.558739 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:32.558696 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:32.619445 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:32.619409 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:43.403856 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:43.403816 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:43.504622 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:43.504588 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:43.559171 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:43.559141 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:49.828113 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:24:49.828075 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:11.114436 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:25:11.114341 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:11.200574 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:25:11.200539 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:11.325710 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:25:11.325673 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:11.630283 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:25:11.630248 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:11.748596 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:25:11.748558 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:11.799780 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:25:11.799750 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:47.152079 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:47.152039 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:47.238904 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:47.238865 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.140156 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.140122 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.274873 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.274834 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.424921 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.424831 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.525781 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.525748 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.644014 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.643981 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.746854 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.746826 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.811042 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.811016 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:58.865905 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:58.865873 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:59.207717 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:59.207627 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:59.399709 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:26:59.399671 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:00.423123 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:27:00.423082 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:31.558720 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:27:31.558679 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:44.092775 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:27:44.092749 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:27:44.097329 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:27:44.097302 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:28:28.398872 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.398839 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft"] Jul 07 02:28:28.401618 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.401601 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.404398 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.404362 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-2346e-serving-cert\"" Jul 07 02:28:28.405574 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.405559 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-2346e-kube-rbac-proxy-sar-config\"" Jul 07 02:28:28.405654 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.405637 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:28:28.405784 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.405771 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gqpvr\"" Jul 07 02:28:28.425847 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.425814 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft"] Jul 07 02:28:28.463221 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.463184 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:28.532565 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.532535 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eefb071-074a-49b8-a411-19b58c077bff-openshift-service-ca-bundle\") pod \"switch-graph-2346e-7cbb7d79b-sfwft\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.532704 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.532572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eefb071-074a-49b8-a411-19b58c077bff-proxy-tls\") pod \"switch-graph-2346e-7cbb7d79b-sfwft\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.633620 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.633582 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eefb071-074a-49b8-a411-19b58c077bff-openshift-service-ca-bundle\") pod \"switch-graph-2346e-7cbb7d79b-sfwft\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.633776 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.633628 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eefb071-074a-49b8-a411-19b58c077bff-proxy-tls\") pod \"switch-graph-2346e-7cbb7d79b-sfwft\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.634326 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.634305 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eefb071-074a-49b8-a411-19b58c077bff-openshift-service-ca-bundle\") pod \"switch-graph-2346e-7cbb7d79b-sfwft\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.636102 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.636079 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eefb071-074a-49b8-a411-19b58c077bff-proxy-tls\") pod \"switch-graph-2346e-7cbb7d79b-sfwft\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.711167 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.711088 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:28.857410 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.856003 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft"] Jul 07 02:28:28.860366 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:28:28.860332 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eefb071_074a_49b8_a411_19b58c077bff.slice/crio-13743e78b8fe13615c6bd90b1b610cb39703136bdf35781c18b8dc3b12c4c0ae WatchSource:0}: Error finding container 13743e78b8fe13615c6bd90b1b610cb39703136bdf35781c18b8dc3b12c4c0ae: Status 404 returned error can't find the container with id 13743e78b8fe13615c6bd90b1b610cb39703136bdf35781c18b8dc3b12c4c0ae Jul 07 02:28:28.862562 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:28.862542 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 02:28:29.840233 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:29.840180 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" event={"ID":"6eefb071-074a-49b8-a411-19b58c077bff","Type":"ContainerStarted","Data":"13743e78b8fe13615c6bd90b1b610cb39703136bdf35781c18b8dc3b12c4c0ae"} Jul 07 02:28:29.935006 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:29.934963 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:31.849697 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:31.849659 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" event={"ID":"6eefb071-074a-49b8-a411-19b58c077bff","Type":"ContainerStarted","Data":"f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe"} Jul 07 02:28:31.850141 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:31.849738 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:37.859196 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:37.859167 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:37.882968 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:37.882913 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podStartSLOduration=7.514817639 podStartE2EDuration="9.882897839s" totalImagesPullingTime="2.3680802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:28:31.871389845 +0000 UTC m=+648.234592426" watchObservedRunningTime="2026-07-07 02:28:37.882897839 +0000 UTC m=+654.246100450" Jul 07 02:28:38.473230 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:38.473189 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft"] Jul 07 02:28:38.473443 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:38.473361 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" containerID="cri-o://f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe" gracePeriod=30 Jul 07 02:28:38.514078 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:38.514045 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:38.700230 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:38.700199 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:38.912808 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:38.912780 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:39.050058 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:39.050024 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:39.190281 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:39.190191 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:42.857280 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:42.857236 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:28:47.858200 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:47.858163 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:28:52.858037 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:52.858000 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:28:52.858567 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:52.858117 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:28:57.857207 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:28:57.857167 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:02.857733 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:02.857693 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:07.857737 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:07.857700 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:08.289663 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.289631 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn"] Jul 07 02:29:08.296098 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.296079 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.298646 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.298623 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 07 02:29:08.299456 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.299421 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 07 02:29:08.300529 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.300504 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:08.306005 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.305982 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn"] Jul 07 02:29:08.329539 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.329515 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls\") pod \"model-chainer-7c56457f57-rzgnn\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.329670 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.329585 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-openshift-service-ca-bundle\") pod \"model-chainer-7c56457f57-rzgnn\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.430858 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.430819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls\") pod \"model-chainer-7c56457f57-rzgnn\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.431062 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.430991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-openshift-service-ca-bundle\") pod \"model-chainer-7c56457f57-rzgnn\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.431062 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:29:08.430997 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Jul 07 02:29:08.431177 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:29:08.431107 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls podName:df9ad0dd-a68c-4b6d-b353-f7d86c6c5439 nodeName:}" failed. No retries permitted until 2026-07-07 02:29:08.93109056 +0000 UTC m=+685.294293121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls") pod "model-chainer-7c56457f57-rzgnn" (UID: "df9ad0dd-a68c-4b6d-b353-f7d86c6c5439") : secret "model-chainer-serving-cert" not found Jul 07 02:29:08.431633 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.431611 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-openshift-service-ca-bundle\") pod \"model-chainer-7c56457f57-rzgnn\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.555847 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.555775 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:08.610367 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.610339 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:29:08.632445 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.632412 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eefb071-074a-49b8-a411-19b58c077bff-openshift-service-ca-bundle\") pod \"6eefb071-074a-49b8-a411-19b58c077bff\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " Jul 07 02:29:08.632610 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.632467 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eefb071-074a-49b8-a411-19b58c077bff-proxy-tls\") pod \"6eefb071-074a-49b8-a411-19b58c077bff\" (UID: \"6eefb071-074a-49b8-a411-19b58c077bff\") " Jul 07 02:29:08.632871 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.632843 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eefb071-074a-49b8-a411-19b58c077bff-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6eefb071-074a-49b8-a411-19b58c077bff" (UID: "6eefb071-074a-49b8-a411-19b58c077bff"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:29:08.635183 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.635154 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eefb071-074a-49b8-a411-19b58c077bff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6eefb071-074a-49b8-a411-19b58c077bff" (UID: "6eefb071-074a-49b8-a411-19b58c077bff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:29:08.733260 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.733218 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eefb071-074a-49b8-a411-19b58c077bff-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:29:08.733260 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.733251 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eefb071-074a-49b8-a411-19b58c077bff-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:29:08.935057 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.934964 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls\") pod \"model-chainer-7c56457f57-rzgnn\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.937453 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.937427 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls\") pod \"model-chainer-7c56457f57-rzgnn\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:08.985816 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.985778 2557 generic.go:356] "Generic (PLEG): container finished" podID="6eefb071-074a-49b8-a411-19b58c077bff" containerID="f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe" exitCode=0 Jul 07 02:29:08.985816 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.985819 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" event={"ID":"6eefb071-074a-49b8-a411-19b58c077bff","Type":"ContainerDied","Data":"f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe"} Jul 07 02:29:08.986023 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.985823 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" Jul 07 02:29:08.986023 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.985885 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft" event={"ID":"6eefb071-074a-49b8-a411-19b58c077bff","Type":"ContainerDied","Data":"13743e78b8fe13615c6bd90b1b610cb39703136bdf35781c18b8dc3b12c4c0ae"} Jul 07 02:29:08.986023 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.985908 2557 scope.go:117] "RemoveContainer" containerID="f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe" Jul 07 02:29:08.994634 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.994608 2557 scope.go:117] "RemoveContainer" containerID="f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe" Jul 07 02:29:08.994920 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:29:08.994898 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe\": container with ID starting with f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe not found: ID does not exist" containerID="f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe" Jul 07 02:29:08.994988 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:08.994929 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe"} err="failed to get container status \"f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe\": rpc error: code = NotFound desc = could not find container \"f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe\": container with ID starting with f56f9682b840ee44e9d92715c93c0d0477a66acab89d8eb48f9678a89fc7dcfe not found: ID does not exist" Jul 07 02:29:09.007173 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:09.007147 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft"] Jul 07 02:29:09.009248 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:09.009225 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2346e-7cbb7d79b-sfwft"] Jul 07 02:29:09.206803 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:09.206709 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:09.357645 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:09.357612 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn"] Jul 07 02:29:09.360842 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:29:09.360814 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9ad0dd_a68c_4b6d_b353_f7d86c6c5439.slice/crio-c87da9b7ad6a53254dfb0e7c67a2f58b56626be8f32c61d25a0f9c5bc58ab9f6 WatchSource:0}: Error finding container c87da9b7ad6a53254dfb0e7c67a2f58b56626be8f32c61d25a0f9c5bc58ab9f6: Status 404 returned error can't find the container with id c87da9b7ad6a53254dfb0e7c67a2f58b56626be8f32c61d25a0f9c5bc58ab9f6 Jul 07 02:29:09.992110 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:09.992072 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" event={"ID":"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439","Type":"ContainerStarted","Data":"061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936"} Jul 07 02:29:09.992512 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:09.992118 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" event={"ID":"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439","Type":"ContainerStarted","Data":"c87da9b7ad6a53254dfb0e7c67a2f58b56626be8f32c61d25a0f9c5bc58ab9f6"} Jul 07 02:29:09.992512 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:09.992211 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:10.012193 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:10.012143 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podStartSLOduration=2.012126085 podStartE2EDuration="2.012126085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:10.011033828 +0000 UTC m=+686.374236424" watchObservedRunningTime="2026-07-07 02:29:10.012126085 +0000 UTC m=+686.375328669" Jul 07 02:29:10.199798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:10.199766 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eefb071-074a-49b8-a411-19b58c077bff" path="/var/lib/kubelet/pods/6eefb071-074a-49b8-a411-19b58c077bff/volumes" Jul 07 02:29:16.001841 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:16.001811 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:18.386044 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.386012 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn"] Jul 07 02:29:18.386484 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.386228 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" containerID="cri-o://061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936" gracePeriod=30 Jul 07 02:29:18.424507 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.424461 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:18.518793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.518753 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:18.613400 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.613345 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:18.702601 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.702503 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:18.783563 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.783531 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:18.884404 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:18.884349 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:21.000077 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:21.000036 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:26.000286 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:26.000248 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:30.999652 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:30.999609 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:31.000218 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:30.999735 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:36.000867 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:36.000825 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:41.000187 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:41.000149 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:46.000367 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:46.000328 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:48.573133 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.573105 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:48.657117 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.657042 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-openshift-service-ca-bundle\") pod \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " Jul 07 02:29:48.657117 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.657091 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls\") pod \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\" (UID: \"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439\") " Jul 07 02:29:48.657475 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.657450 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" (UID: "df9ad0dd-a68c-4b6d-b353-f7d86c6c5439"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:29:48.659265 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.659240 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" (UID: "df9ad0dd-a68c-4b6d-b353-f7d86c6c5439"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:29:48.758612 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.758574 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:29:48.758612 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.758603 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:29:48.777686 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.777661 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql"] Jul 07 02:29:48.777941 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.777929 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" Jul 07 02:29:48.777992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.777943 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" Jul 07 02:29:48.777992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.777964 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" Jul 07 02:29:48.777992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.777970 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" Jul 07 02:29:48.778090 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.778012 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eefb071-074a-49b8-a411-19b58c077bff" containerName="switch-graph-2346e" Jul 07 02:29:48.778090 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.778020 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerName="model-chainer" Jul 07 02:29:48.780760 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.780743 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:48.783439 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.783417 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3ea7f-kube-rbac-proxy-sar-config\"" Jul 07 02:29:48.783546 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.783428 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3ea7f-serving-cert\"" Jul 07 02:29:48.784604 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.784561 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:48.792639 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.792616 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql"] Jul 07 02:29:48.859729 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.859694 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4665b529-1d20-4dd1-b943-69a2a65c200a-proxy-tls\") pod \"switch-graph-3ea7f-9cf7d776c-xhlql\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:48.859871 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.859736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4665b529-1d20-4dd1-b943-69a2a65c200a-openshift-service-ca-bundle\") pod \"switch-graph-3ea7f-9cf7d776c-xhlql\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:48.899596 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.899560 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:48.960368 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.960297 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4665b529-1d20-4dd1-b943-69a2a65c200a-proxy-tls\") pod \"switch-graph-3ea7f-9cf7d776c-xhlql\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:48.960368 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.960340 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4665b529-1d20-4dd1-b943-69a2a65c200a-openshift-service-ca-bundle\") pod \"switch-graph-3ea7f-9cf7d776c-xhlql\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:48.960937 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.960916 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4665b529-1d20-4dd1-b943-69a2a65c200a-openshift-service-ca-bundle\") pod \"switch-graph-3ea7f-9cf7d776c-xhlql\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:48.962688 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:48.962671 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4665b529-1d20-4dd1-b943-69a2a65c200a-proxy-tls\") pod \"switch-graph-3ea7f-9cf7d776c-xhlql\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:49.091009 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.090958 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:49.140882 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.140846 2557 generic.go:356] "Generic (PLEG): container finished" podID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" containerID="061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936" exitCode=0 Jul 07 02:29:49.141050 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.140904 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" event={"ID":"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439","Type":"ContainerDied","Data":"061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936"} Jul 07 02:29:49.141050 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.140956 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" event={"ID":"df9ad0dd-a68c-4b6d-b353-f7d86c6c5439","Type":"ContainerDied","Data":"c87da9b7ad6a53254dfb0e7c67a2f58b56626be8f32c61d25a0f9c5bc58ab9f6"} Jul 07 02:29:49.141050 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.140989 2557 scope.go:117] "RemoveContainer" containerID="061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936" Jul 07 02:29:49.141477 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.140909 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn" Jul 07 02:29:49.158320 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.156420 2557 scope.go:117] "RemoveContainer" containerID="061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936" Jul 07 02:29:49.158320 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:29:49.156815 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936\": container with ID starting with 061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936 not found: ID does not exist" containerID="061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936" Jul 07 02:29:49.158320 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.156854 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936"} err="failed to get container status \"061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936\": rpc error: code = NotFound desc = could not find container \"061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936\": container with ID starting with 061f0bbbb4a0e2d9955afad1ef099d76da7d6c85503fe479098f4f87955ff936 not found: ID does not exist" Jul 07 02:29:49.175296 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.175266 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn"] Jul 07 02:29:49.178298 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.178273 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7c56457f57-rzgnn"] Jul 07 02:29:49.225534 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:49.225511 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql"] Jul 07 02:29:49.227917 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:29:49.227884 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4665b529_1d20_4dd1_b943_69a2a65c200a.slice/crio-1ec0d672fde270b3750aa5d8b8198dd61a837e0d8461b91eeca7b2ff86ff5534 WatchSource:0}: Error finding container 1ec0d672fde270b3750aa5d8b8198dd61a837e0d8461b91eeca7b2ff86ff5534: Status 404 returned error can't find the container with id 1ec0d672fde270b3750aa5d8b8198dd61a837e0d8461b91eeca7b2ff86ff5534 Jul 07 02:29:50.146259 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:50.146213 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" event={"ID":"4665b529-1d20-4dd1-b943-69a2a65c200a","Type":"ContainerStarted","Data":"613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd"} Jul 07 02:29:50.146994 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:50.146284 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:29:50.146994 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:50.146306 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" event={"ID":"4665b529-1d20-4dd1-b943-69a2a65c200a","Type":"ContainerStarted","Data":"1ec0d672fde270b3750aa5d8b8198dd61a837e0d8461b91eeca7b2ff86ff5534"} Jul 07 02:29:50.163883 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:50.163836 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podStartSLOduration=2.163821211 podStartE2EDuration="2.163821211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:50.161886929 +0000 UTC m=+726.525089514" watchObservedRunningTime="2026-07-07 02:29:50.163821211 +0000 UTC m=+726.527023794" Jul 07 02:29:50.199184 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:50.199153 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9ad0dd-a68c-4b6d-b353-f7d86c6c5439" path="/var/lib/kubelet/pods/df9ad0dd-a68c-4b6d-b353-f7d86c6c5439/volumes" Jul 07 02:29:56.156682 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:29:56.156649 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:30:01.925216 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:01.925179 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:18.609096 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.609013 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd"] Jul 07 02:30:18.612215 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.612193 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:18.615589 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.615568 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-821ea-serving-cert\"" Jul 07 02:30:18.615681 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.615633 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-821ea-kube-rbac-proxy-sar-config\"" Jul 07 02:30:18.623186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.623164 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd"] Jul 07 02:30:18.641630 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.641597 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:18.697842 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.697804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls\") pod \"sequence-graph-821ea-f6c64f499-77srd\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:18.698049 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.697862 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560669b5-f71a-48ed-bbf4-d1d6036b3341-openshift-service-ca-bundle\") pod \"sequence-graph-821ea-f6c64f499-77srd\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:18.798945 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.798897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560669b5-f71a-48ed-bbf4-d1d6036b3341-openshift-service-ca-bundle\") pod \"sequence-graph-821ea-f6c64f499-77srd\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:18.799121 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.798977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls\") pod \"sequence-graph-821ea-f6c64f499-77srd\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:18.799121 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:30:18.799070 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-821ea-serving-cert: secret "sequence-graph-821ea-serving-cert" not found Jul 07 02:30:18.799121 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:30:18.799122 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls podName:560669b5-f71a-48ed-bbf4-d1d6036b3341 nodeName:}" failed. No retries permitted until 2026-07-07 02:30:19.29910715 +0000 UTC m=+755.662309712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls") pod "sequence-graph-821ea-f6c64f499-77srd" (UID: "560669b5-f71a-48ed-bbf4-d1d6036b3341") : secret "sequence-graph-821ea-serving-cert" not found Jul 07 02:30:18.799588 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.799567 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560669b5-f71a-48ed-bbf4-d1d6036b3341-openshift-service-ca-bundle\") pod \"sequence-graph-821ea-f6c64f499-77srd\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:18.935316 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:18.935205 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:19.302262 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:19.302230 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls\") pod \"sequence-graph-821ea-f6c64f499-77srd\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:19.304599 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:19.304575 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls\") pod \"sequence-graph-821ea-f6c64f499-77srd\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:19.522427 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:19.522357 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:19.649370 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:19.649328 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd"] Jul 07 02:30:19.651580 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:30:19.651554 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560669b5_f71a_48ed_bbf4_d1d6036b3341.slice/crio-9a2775efbf6f1e561c917f6346a4ac001b0ca70a0e4dd16300726c407d742a63 WatchSource:0}: Error finding container 9a2775efbf6f1e561c917f6346a4ac001b0ca70a0e4dd16300726c407d742a63: Status 404 returned error can't find the container with id 9a2775efbf6f1e561c917f6346a4ac001b0ca70a0e4dd16300726c407d742a63 Jul 07 02:30:20.261594 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:20.261555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" event={"ID":"560669b5-f71a-48ed-bbf4-d1d6036b3341","Type":"ContainerStarted","Data":"ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb"} Jul 07 02:30:20.261798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:20.261613 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" event={"ID":"560669b5-f71a-48ed-bbf4-d1d6036b3341","Type":"ContainerStarted","Data":"9a2775efbf6f1e561c917f6346a4ac001b0ca70a0e4dd16300726c407d742a63"} Jul 07 02:30:20.261798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:20.261663 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:20.279622 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:20.279573 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podStartSLOduration=2.2795572330000002 podStartE2EDuration="2.279557233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:20.278057598 +0000 UTC m=+756.641260182" watchObservedRunningTime="2026-07-07 02:30:20.279557233 +0000 UTC m=+756.642759815" Jul 07 02:30:26.273840 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:26.273811 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:30:51.641638 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:51.641604 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:51.704486 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:30:51.704457 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:15.141762 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:31:15.141669 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:44.117140 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:32:44.117113 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:32:44.119309 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:32:44.119285 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:34:40.638445 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:34:40.638344 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:36:54.330311 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:36:54.330279 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:37:44.142545 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:37:44.142515 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:37:44.143052 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:37:44.142515 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:38:03.401645 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:03.401606 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql"] Jul 07 02:38:03.402170 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:03.401826 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" containerID="cri-o://613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd" gracePeriod=30 Jul 07 02:38:03.428515 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:03.428479 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:03.551558 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:03.551522 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:03.635360 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:03.635315 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:03.706321 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:03.706226 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:03.922456 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:03.922427 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:04.225791 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:04.225758 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:06.155508 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:06.155469 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:11.155087 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:11.155049 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:16.155837 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:16.155800 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:16.156312 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:16.155909 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:38:21.154920 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:21.154881 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:26.154949 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:26.154913 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:31.155026 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:31.154990 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:33.406157 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.406024 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd"] Jul 07 02:38:33.406532 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.406238 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" containerID="cri-o://ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb" gracePeriod=30 Jul 07 02:38:33.442073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.442030 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:33.544046 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.544014 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:33.557567 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.557543 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:38:33.635903 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.635871 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4665b529-1d20-4dd1-b943-69a2a65c200a-openshift-service-ca-bundle\") pod \"4665b529-1d20-4dd1-b943-69a2a65c200a\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " Jul 07 02:38:33.636103 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.635948 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4665b529-1d20-4dd1-b943-69a2a65c200a-proxy-tls\") pod \"4665b529-1d20-4dd1-b943-69a2a65c200a\" (UID: \"4665b529-1d20-4dd1-b943-69a2a65c200a\") " Jul 07 02:38:33.636238 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.636214 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4665b529-1d20-4dd1-b943-69a2a65c200a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4665b529-1d20-4dd1-b943-69a2a65c200a" (UID: "4665b529-1d20-4dd1-b943-69a2a65c200a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:38:33.637940 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.637912 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4665b529-1d20-4dd1-b943-69a2a65c200a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4665b529-1d20-4dd1-b943-69a2a65c200a" (UID: "4665b529-1d20-4dd1-b943-69a2a65c200a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:38:33.641005 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.640979 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:33.733045 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.732970 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:33.737062 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.737039 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4665b529-1d20-4dd1-b943-69a2a65c200a-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:38:33.737117 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.737066 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4665b529-1d20-4dd1-b943-69a2a65c200a-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:38:33.927872 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:33.927835 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:34.027940 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.027909 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:34.152846 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.152813 2557 generic.go:356] "Generic (PLEG): container finished" podID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerID="613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd" exitCode=0 Jul 07 02:38:34.153036 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.152888 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" event={"ID":"4665b529-1d20-4dd1-b943-69a2a65c200a","Type":"ContainerDied","Data":"613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd"} Jul 07 02:38:34.153036 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.152893 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" Jul 07 02:38:34.153036 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.152970 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql" event={"ID":"4665b529-1d20-4dd1-b943-69a2a65c200a","Type":"ContainerDied","Data":"1ec0d672fde270b3750aa5d8b8198dd61a837e0d8461b91eeca7b2ff86ff5534"} Jul 07 02:38:34.153036 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.153000 2557 scope.go:117] "RemoveContainer" containerID="613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd" Jul 07 02:38:34.162357 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.162341 2557 scope.go:117] "RemoveContainer" containerID="613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd" Jul 07 02:38:34.162657 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:38:34.162638 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd\": container with ID starting with 613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd not found: ID does not exist" containerID="613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd" Jul 07 02:38:34.162714 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.162667 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd"} err="failed to get container status \"613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd\": rpc error: code = NotFound desc = could not find container \"613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd\": container with ID starting with 613cbcbde51d5ac03ca4adf2a90cd2ce8944842b92d6efa8f44724a38e7aa3bd not found: ID does not exist" Jul 07 02:38:34.175713 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.175692 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql"] Jul 07 02:38:34.179443 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.179420 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3ea7f-9cf7d776c-xhlql"] Jul 07 02:38:34.198774 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:34.198748 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" path="/var/lib/kubelet/pods/4665b529-1d20-4dd1-b943-69a2a65c200a/volumes" Jul 07 02:38:36.272613 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:36.272530 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:40.524151 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:40.524107 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:41.272092 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:41.272056 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:46.272808 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:46.272763 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:46.273214 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:46.272859 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:38:51.272166 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:51.272126 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:38:56.271920 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:38:56.271882 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:01.272735 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:01.272694 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:03.550996 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.550974 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:39:03.662548 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.662464 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz"] Jul 07 02:39:03.662827 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.662809 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" Jul 07 02:39:03.662963 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.662829 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" Jul 07 02:39:03.662963 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.662857 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" Jul 07 02:39:03.662963 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.662866 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" Jul 07 02:39:03.662963 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.662952 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerName="sequence-graph-821ea" Jul 07 02:39:03.663158 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.662966 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4665b529-1d20-4dd1-b943-69a2a65c200a" containerName="switch-graph-3ea7f" Jul 07 02:39:03.665791 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.665760 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:03.666420 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.666393 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:03.668233 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.668212 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cec47-kube-rbac-proxy-sar-config\"" Jul 07 02:39:03.668318 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.668220 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cec47-serving-cert\"" Jul 07 02:39:03.669997 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.669977 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls\") pod \"560669b5-f71a-48ed-bbf4-d1d6036b3341\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " Jul 07 02:39:03.670106 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.670090 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560669b5-f71a-48ed-bbf4-d1d6036b3341-openshift-service-ca-bundle\") pod \"560669b5-f71a-48ed-bbf4-d1d6036b3341\" (UID: \"560669b5-f71a-48ed-bbf4-d1d6036b3341\") " Jul 07 02:39:03.670446 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.670423 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560669b5-f71a-48ed-bbf4-d1d6036b3341-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "560669b5-f71a-48ed-bbf4-d1d6036b3341" (UID: "560669b5-f71a-48ed-bbf4-d1d6036b3341"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:39:03.673177 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.672979 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "560669b5-f71a-48ed-bbf4-d1d6036b3341" (UID: "560669b5-f71a-48ed-bbf4-d1d6036b3341"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:39:03.674499 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.674477 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz"] Jul 07 02:39:03.769898 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.769870 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:03.770938 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.770916 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-proxy-tls\") pod \"ensemble-graph-cec47-67f6fbdc44-f52tz\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:03.771060 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.771039 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-openshift-service-ca-bundle\") pod \"ensemble-graph-cec47-67f6fbdc44-f52tz\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:03.771111 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.771103 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/560669b5-f71a-48ed-bbf4-d1d6036b3341-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:39:03.771143 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.771115 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560669b5-f71a-48ed-bbf4-d1d6036b3341-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:39:03.871662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.871629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-openshift-service-ca-bundle\") pod \"ensemble-graph-cec47-67f6fbdc44-f52tz\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:03.871662 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.871670 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-proxy-tls\") pod \"ensemble-graph-cec47-67f6fbdc44-f52tz\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:03.872350 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.872322 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-openshift-service-ca-bundle\") pod \"ensemble-graph-cec47-67f6fbdc44-f52tz\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:03.874073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.874053 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-proxy-tls\") pod \"ensemble-graph-cec47-67f6fbdc44-f52tz\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:03.984108 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:03.984029 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:04.110332 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.110308 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz"] Jul 07 02:39:04.112861 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:39:04.112839 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450 WatchSource:0}: Error finding container 9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450: Status 404 returned error can't find the container with id 9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450 Jul 07 02:39:04.272082 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.272052 2557 generic.go:356] "Generic (PLEG): container finished" podID="560669b5-f71a-48ed-bbf4-d1d6036b3341" containerID="ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb" exitCode=0 Jul 07 02:39:04.272250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.272110 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" Jul 07 02:39:04.272250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.272130 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" event={"ID":"560669b5-f71a-48ed-bbf4-d1d6036b3341","Type":"ContainerDied","Data":"ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb"} Jul 07 02:39:04.272250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.272207 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd" event={"ID":"560669b5-f71a-48ed-bbf4-d1d6036b3341","Type":"ContainerDied","Data":"9a2775efbf6f1e561c917f6346a4ac001b0ca70a0e4dd16300726c407d742a63"} Jul 07 02:39:04.272250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.272237 2557 scope.go:117] "RemoveContainer" containerID="ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb" Jul 07 02:39:04.273808 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.273784 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" event={"ID":"50f3dd6b-32b0-410e-b1fa-bda78c95fe20","Type":"ContainerStarted","Data":"d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e"} Jul 07 02:39:04.273923 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.273832 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" event={"ID":"50f3dd6b-32b0-410e-b1fa-bda78c95fe20","Type":"ContainerStarted","Data":"9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450"} Jul 07 02:39:04.273923 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.273860 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:04.280273 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.280258 2557 scope.go:117] "RemoveContainer" containerID="ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb" Jul 07 02:39:04.280577 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:39:04.280558 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb\": container with ID starting with ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb not found: ID does not exist" containerID="ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb" Jul 07 02:39:04.280639 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.280584 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb"} err="failed to get container status \"ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb\": rpc error: code = NotFound desc = could not find container \"ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb\": container with ID starting with ad10d2d8df7b4c1de1982398da3de99fb8123227b108cd007df124e865d4a4fb not found: ID does not exist" Jul 07 02:39:04.289805 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.289783 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd"] Jul 07 02:39:04.292054 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.292034 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-821ea-f6c64f499-77srd"] Jul 07 02:39:04.307658 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:04.307620 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podStartSLOduration=1.307605837 podStartE2EDuration="1.307605837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:39:04.306149326 +0000 UTC m=+1280.669351912" watchObservedRunningTime="2026-07-07 02:39:04.307605837 +0000 UTC m=+1280.670808453" Jul 07 02:39:06.199350 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:06.199314 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560669b5-f71a-48ed-bbf4-d1d6036b3341" path="/var/lib/kubelet/pods/560669b5-f71a-48ed-bbf4-d1d6036b3341/volumes" Jul 07 02:39:10.284670 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:10.284644 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:13.732809 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:13.732736 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz"] Jul 07 02:39:13.733190 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:13.732912 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" containerID="cri-o://d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e" gracePeriod=30 Jul 07 02:39:13.757498 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:13.757473 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:13.852939 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:13.852908 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:13.926451 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:13.926416 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:14.033471 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:14.033442 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:14.214743 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:14.214717 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:14.410676 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:14.410593 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:15.283021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:15.282983 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:20.282498 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:20.282455 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:25.282754 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:25.282710 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:25.283264 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:25.282860 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:30.282129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:30.282086 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:33.583992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.583933 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx"] Jul 07 02:39:33.587103 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.587085 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:33.589647 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.589624 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-524cd-kube-rbac-proxy-sar-config\"" Jul 07 02:39:33.589768 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.589624 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-524cd-serving-cert\"" Jul 07 02:39:33.590649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.590616 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:33.595155 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.595132 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-openshift-service-ca-bundle\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:33.595267 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.595243 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:33.595928 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.595909 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx"] Jul 07 02:39:33.696609 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.696567 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-openshift-service-ca-bundle\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:33.696783 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.696623 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:33.696783 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:39:33.696755 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-524cd-serving-cert: secret "sequence-graph-524cd-serving-cert" not found Jul 07 02:39:33.696859 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:39:33.696812 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls podName:fe76f6bf-5c18-4c9d-8142-47c5318c59d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:39:34.196794765 +0000 UTC m=+1310.559997326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls") pod "sequence-graph-524cd-55f58ffb86-hvhqx" (UID: "fe76f6bf-5c18-4c9d-8142-47c5318c59d3") : secret "sequence-graph-524cd-serving-cert" not found Jul 07 02:39:33.697233 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:33.697212 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-openshift-service-ca-bundle\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:34.199366 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:34.199332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:34.199545 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:39:34.199469 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-524cd-serving-cert: secret "sequence-graph-524cd-serving-cert" not found Jul 07 02:39:34.199545 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:39:34.199526 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls podName:fe76f6bf-5c18-4c9d-8142-47c5318c59d3 nodeName:}" failed. No retries permitted until 2026-07-07 02:39:35.199512127 +0000 UTC m=+1311.562714692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls") pod "sequence-graph-524cd-55f58ffb86-hvhqx" (UID: "fe76f6bf-5c18-4c9d-8142-47c5318c59d3") : secret "sequence-graph-524cd-serving-cert" not found Jul 07 02:39:34.602938 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:34.602901 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:35.205778 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:35.205729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:35.208122 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:35.208099 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls\") pod \"sequence-graph-524cd-55f58ffb86-hvhqx\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:35.282547 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:35.282511 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:35.398445 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:35.398419 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:35.522177 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:35.522145 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx"] Jul 07 02:39:35.524901 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:39:35.524864 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe76f6bf_5c18_4c9d_8142_47c5318c59d3.slice/crio-36c943ba97d40fda6acc60f11ee0da2439859abf5b77ca2164dc3c14c0a79cf3 WatchSource:0}: Error finding container 36c943ba97d40fda6acc60f11ee0da2439859abf5b77ca2164dc3c14c0a79cf3: Status 404 returned error can't find the container with id 36c943ba97d40fda6acc60f11ee0da2439859abf5b77ca2164dc3c14c0a79cf3 Jul 07 02:39:36.393156 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:36.393118 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" event={"ID":"fe76f6bf-5c18-4c9d-8142-47c5318c59d3","Type":"ContainerStarted","Data":"4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764"} Jul 07 02:39:36.393547 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:36.393178 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" event={"ID":"fe76f6bf-5c18-4c9d-8142-47c5318c59d3","Type":"ContainerStarted","Data":"36c943ba97d40fda6acc60f11ee0da2439859abf5b77ca2164dc3c14c0a79cf3"} Jul 07 02:39:36.393547 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:36.393205 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:36.410287 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:36.410239 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podStartSLOduration=3.41022534 podStartE2EDuration="3.41022534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:39:36.409212043 +0000 UTC m=+1312.772414627" watchObservedRunningTime="2026-07-07 02:39:36.41022534 +0000 UTC m=+1312.773427927" Jul 07 02:39:40.282250 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:40.282215 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:42.408718 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:42.408682 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:39:43.670736 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.670700 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx"] Jul 07 02:39:43.671104 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.670906 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" containerID="cri-o://4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764" gracePeriod=30 Jul 07 02:39:43.696634 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.696604 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:43.919754 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.919730 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:43.973681 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.973608 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-openshift-service-ca-bundle\") pod \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " Jul 07 02:39:43.973816 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.973702 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-proxy-tls\") pod \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\" (UID: \"50f3dd6b-32b0-410e-b1fa-bda78c95fe20\") " Jul 07 02:39:43.973997 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.973976 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "50f3dd6b-32b0-410e-b1fa-bda78c95fe20" (UID: "50f3dd6b-32b0-410e-b1fa-bda78c95fe20"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:39:43.975611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:43.975588 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50f3dd6b-32b0-410e-b1fa-bda78c95fe20" (UID: "50f3dd6b-32b0-410e-b1fa-bda78c95fe20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:39:44.074193 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.074165 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:39:44.074193 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.074188 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f3dd6b-32b0-410e-b1fa-bda78c95fe20-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:39:44.431162 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.431128 2557 generic.go:356] "Generic (PLEG): container finished" podID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerID="d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e" exitCode=137 Jul 07 02:39:44.431324 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.431187 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" Jul 07 02:39:44.431324 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.431234 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" event={"ID":"50f3dd6b-32b0-410e-b1fa-bda78c95fe20","Type":"ContainerDied","Data":"d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e"} Jul 07 02:39:44.431324 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.431275 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz" event={"ID":"50f3dd6b-32b0-410e-b1fa-bda78c95fe20","Type":"ContainerDied","Data":"9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450"} Jul 07 02:39:44.431324 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.431299 2557 scope.go:117] "RemoveContainer" containerID="d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e" Jul 07 02:39:44.438672 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.438655 2557 scope.go:117] "RemoveContainer" containerID="d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e" Jul 07 02:39:44.440686 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:39:44.440662 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e\": container with ID starting with d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e not found: ID does not exist" containerID="d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e" Jul 07 02:39:44.440798 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.440690 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e"} err="failed to get container status \"d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e\": rpc error: code = NotFound desc = could not find container \"d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e\": container with ID starting with d066a41a78e7028cf420a3d683a6800acee09bb2c3e8ec4330a741d64d0f450e not found: ID does not exist" Jul 07 02:39:44.447424 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.447401 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz"] Jul 07 02:39:44.451023 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:44.451002 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cec47-67f6fbdc44-f52tz"] Jul 07 02:39:45.102080 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:45.102048 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:46.199514 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:46.199480 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" path="/var/lib/kubelet/pods/50f3dd6b-32b0-410e-b1fa-bda78c95fe20/volumes" Jul 07 02:39:47.407957 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:47.407916 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:52.407279 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:52.407241 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:57.407067 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:57.407026 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:39:57.407632 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:39:57.407133 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:40:02.407205 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:02.407167 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:40:07.407276 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:07.407194 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:40:12.407011 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:12.406974 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:40:13.810124 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:13.810096 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:40:13.899509 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:13.899465 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-openshift-service-ca-bundle\") pod \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " Jul 07 02:40:13.899509 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:13.899518 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls\") pod \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\" (UID: \"fe76f6bf-5c18-4c9d-8142-47c5318c59d3\") " Jul 07 02:40:13.899822 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:13.899797 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fe76f6bf-5c18-4c9d-8142-47c5318c59d3" (UID: "fe76f6bf-5c18-4c9d-8142-47c5318c59d3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:40:13.901551 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:13.901528 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe76f6bf-5c18-4c9d-8142-47c5318c59d3" (UID: "fe76f6bf-5c18-4c9d-8142-47c5318c59d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:40:14.000182 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.000149 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:40:14.000182 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.000178 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe76f6bf-5c18-4c9d-8142-47c5318c59d3-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:40:14.005457 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.005432 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh"] Jul 07 02:40:14.005790 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.005773 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" Jul 07 02:40:14.005861 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.005793 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" Jul 07 02:40:14.005861 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.005815 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" Jul 07 02:40:14.005861 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.005824 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" Jul 07 02:40:14.006012 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.005900 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerName="sequence-graph-524cd" Jul 07 02:40:14.006012 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.005917 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f3dd6b-32b0-410e-b1fa-bda78c95fe20" containerName="ensemble-graph-cec47" Jul 07 02:40:14.008731 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.008712 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.011364 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.011345 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e4a8c-kube-rbac-proxy-sar-config\"" Jul 07 02:40:14.011477 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.011350 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e4a8c-serving-cert\"" Jul 07 02:40:14.022097 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.022070 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh"] Jul 07 02:40:14.023550 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.023524 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:14.100748 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.100715 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls\") pod \"ensemble-graph-e4a8c-d5b8f5585-rcdfh\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.100945 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.100760 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc200ee-7d9d-43db-8135-ec9e90d38abe-openshift-service-ca-bundle\") pod \"ensemble-graph-e4a8c-d5b8f5585-rcdfh\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.201092 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.201067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls\") pod \"ensemble-graph-e4a8c-d5b8f5585-rcdfh\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.201274 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.201103 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc200ee-7d9d-43db-8135-ec9e90d38abe-openshift-service-ca-bundle\") pod \"ensemble-graph-e4a8c-d5b8f5585-rcdfh\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.201274 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:14.201238 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-e4a8c-serving-cert: secret "ensemble-graph-e4a8c-serving-cert" not found Jul 07 02:40:14.201346 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:14.201291 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls podName:1dc200ee-7d9d-43db-8135-ec9e90d38abe nodeName:}" failed. No retries permitted until 2026-07-07 02:40:14.701276856 +0000 UTC m=+1351.064479423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls") pod "ensemble-graph-e4a8c-d5b8f5585-rcdfh" (UID: "1dc200ee-7d9d-43db-8135-ec9e90d38abe") : secret "ensemble-graph-e4a8c-serving-cert" not found Jul 07 02:40:14.201816 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.201795 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc200ee-7d9d-43db-8135-ec9e90d38abe-openshift-service-ca-bundle\") pod \"ensemble-graph-e4a8c-d5b8f5585-rcdfh\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.286339 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.286271 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:14.540198 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.540118 2557 generic.go:356] "Generic (PLEG): container finished" podID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" containerID="4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764" exitCode=0 Jul 07 02:40:14.540198 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.540175 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" Jul 07 02:40:14.540198 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.540182 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" event={"ID":"fe76f6bf-5c18-4c9d-8142-47c5318c59d3","Type":"ContainerDied","Data":"4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764"} Jul 07 02:40:14.540435 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.540224 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx" event={"ID":"fe76f6bf-5c18-4c9d-8142-47c5318c59d3","Type":"ContainerDied","Data":"36c943ba97d40fda6acc60f11ee0da2439859abf5b77ca2164dc3c14c0a79cf3"} Jul 07 02:40:14.540435 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.540251 2557 scope.go:117] "RemoveContainer" containerID="4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764" Jul 07 02:40:14.547819 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.547797 2557 scope.go:117] "RemoveContainer" containerID="4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764" Jul 07 02:40:14.548065 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:14.548048 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764\": container with ID starting with 4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764 not found: ID does not exist" containerID="4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764" Jul 07 02:40:14.548121 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.548074 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764"} err="failed to get container status \"4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764\": rpc error: code = NotFound desc = could not find container \"4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764\": container with ID starting with 4fd21c927813f109532771937c627feeed31e706b101ae621313e48766671764 not found: ID does not exist" Jul 07 02:40:14.558240 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.558218 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx"] Jul 07 02:40:14.564045 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.564026 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-524cd-55f58ffb86-hvhqx"] Jul 07 02:40:14.704846 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.704814 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls\") pod \"ensemble-graph-e4a8c-d5b8f5585-rcdfh\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.707134 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.707112 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls\") pod \"ensemble-graph-e4a8c-d5b8f5585-rcdfh\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:14.923779 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:14.923680 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:15.046736 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:15.046712 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh"] Jul 07 02:40:15.049137 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:40:15.049105 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc200ee_7d9d_43db_8135_ec9e90d38abe.slice/crio-9c440c5f60d108e1ddf1ce7955b9fed92b803581a37f114a4556206f7171d016 WatchSource:0}: Error finding container 9c440c5f60d108e1ddf1ce7955b9fed92b803581a37f114a4556206f7171d016: Status 404 returned error can't find the container with id 9c440c5f60d108e1ddf1ce7955b9fed92b803581a37f114a4556206f7171d016 Jul 07 02:40:15.545661 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:15.545627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" event={"ID":"1dc200ee-7d9d-43db-8135-ec9e90d38abe","Type":"ContainerStarted","Data":"d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60"} Jul 07 02:40:15.545871 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:15.545685 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:15.545871 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:15.545701 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" event={"ID":"1dc200ee-7d9d-43db-8135-ec9e90d38abe","Type":"ContainerStarted","Data":"9c440c5f60d108e1ddf1ce7955b9fed92b803581a37f114a4556206f7171d016"} Jul 07 02:40:15.563199 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:15.563153 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podStartSLOduration=2.563137399 podStartE2EDuration="2.563137399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:40:15.562539218 +0000 UTC m=+1351.925741801" watchObservedRunningTime="2026-07-07 02:40:15.563137399 +0000 UTC m=+1351.926339981" Jul 07 02:40:16.199029 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:16.198996 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe76f6bf-5c18-4c9d-8142-47c5318c59d3" path="/var/lib/kubelet/pods/fe76f6bf-5c18-4c9d-8142-47c5318c59d3/volumes" Jul 07 02:40:17.494094 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:17.494055 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450\": RecentStats: unable to find data in memory cache]" Jul 07 02:40:21.555021 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:21.554989 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:40:23.887981 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:23.887949 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450\": RecentStats: unable to find data in memory cache]" Jul 07 02:40:23.887981 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:23.887968 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450\": RecentStats: unable to find data in memory cache]" Jul 07 02:40:24.898824 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:24.898782 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450\": RecentStats: unable to find data in memory cache]" Jul 07 02:40:27.502392 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:27.502338 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450\": RecentStats: unable to find data in memory cache]" Jul 07 02:40:37.538805 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:37.538769 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450\": RecentStats: unable to find data in memory cache]" Jul 07 02:40:39.878849 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:39.878805 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f3dd6b_32b0_410e_b1fa_bda78c95fe20.slice/crio-9fcedbfb306e00d3c2df82b75c46877fdb743f697e3b440504cfbeb9a51f0450\": RecentStats: unable to find data in memory cache]" Jul 07 02:40:43.929283 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.929243 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd"] Jul 07 02:40:43.932581 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.932562 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:43.935004 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.934971 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-7c238-kube-rbac-proxy-sar-config\"" Jul 07 02:40:43.935141 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.935042 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-7c238-serving-cert\"" Jul 07 02:40:43.941228 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.941191 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd"] Jul 07 02:40:43.942915 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.942892 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:43.944954 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.944932 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9605e293-a6b5-44a0-880b-04bd9050d607-openshift-service-ca-bundle\") pod \"sequence-graph-7c238-5c449cbc68-zsbrd\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:43.945062 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:43.944974 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls\") pod \"sequence-graph-7c238-5c449cbc68-zsbrd\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:44.046073 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.046037 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9605e293-a6b5-44a0-880b-04bd9050d607-openshift-service-ca-bundle\") pod \"sequence-graph-7c238-5c449cbc68-zsbrd\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:44.046285 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.046087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls\") pod \"sequence-graph-7c238-5c449cbc68-zsbrd\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:44.046285 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:44.046204 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-7c238-serving-cert: secret "sequence-graph-7c238-serving-cert" not found Jul 07 02:40:44.046285 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:40:44.046273 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls podName:9605e293-a6b5-44a0-880b-04bd9050d607 nodeName:}" failed. No retries permitted until 2026-07-07 02:40:44.546252215 +0000 UTC m=+1380.909454799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls") pod "sequence-graph-7c238-5c449cbc68-zsbrd" (UID: "9605e293-a6b5-44a0-880b-04bd9050d607") : secret "sequence-graph-7c238-serving-cert" not found Jul 07 02:40:44.046738 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.046717 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9605e293-a6b5-44a0-880b-04bd9050d607-openshift-service-ca-bundle\") pod \"sequence-graph-7c238-5c449cbc68-zsbrd\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:44.205829 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.205801 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:44.550677 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.550644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls\") pod \"sequence-graph-7c238-5c449cbc68-zsbrd\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:44.553095 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.553073 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls\") pod \"sequence-graph-7c238-5c449cbc68-zsbrd\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:44.843318 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.843222 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:44.979582 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:44.979550 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd"] Jul 07 02:40:44.982723 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:40:44.982698 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9605e293_a6b5_44a0_880b_04bd9050d607.slice/crio-993b30c9ecbb73b073bceea75936327649560d788de66af02732940e99531c67 WatchSource:0}: Error finding container 993b30c9ecbb73b073bceea75936327649560d788de66af02732940e99531c67: Status 404 returned error can't find the container with id 993b30c9ecbb73b073bceea75936327649560d788de66af02732940e99531c67 Jul 07 02:40:45.659575 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:45.659538 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" event={"ID":"9605e293-a6b5-44a0-880b-04bd9050d607","Type":"ContainerStarted","Data":"9fe56fd5946eda8c244ef916b336743f3266aee5a2d4b72d72fc38794465fcad"} Jul 07 02:40:45.659796 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:45.659601 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" event={"ID":"9605e293-a6b5-44a0-880b-04bd9050d607","Type":"ContainerStarted","Data":"993b30c9ecbb73b073bceea75936327649560d788de66af02732940e99531c67"} Jul 07 02:40:45.659796 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:45.659631 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:40:45.678416 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:45.678347 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podStartSLOduration=2.678327656 podStartE2EDuration="2.678327656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:40:45.676670715 +0000 UTC m=+1382.039873302" watchObservedRunningTime="2026-07-07 02:40:45.678327656 +0000 UTC m=+1382.041530239" Jul 07 02:40:51.675403 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:40:51.675354 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:42:44.167647 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:42:44.167614 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:42:44.169042 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:42:44.169018 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:43:22.189613 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:43:22.189582 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:47:44.189988 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:47:44.189960 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:47:44.192232 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:47:44.192205 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:48:28.751003 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:28.750963 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh"] Jul 07 02:48:28.751584 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:28.751219 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" containerID="cri-o://d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60" gracePeriod=30 Jul 07 02:48:28.799660 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:28.799628 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:29.966204 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:29.966163 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:31.554746 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:31.554705 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:48:36.554749 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:36.554701 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:48:41.554524 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:41.554487 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:48:41.554922 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:41.554589 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:48:46.554548 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:46.554508 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:48:51.554104 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:51.554061 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:48:56.554521 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:56.554478 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:48:58.607793 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.607758 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd"] Jul 07 02:48:58.608247 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.608002 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" containerID="cri-o://9fe56fd5946eda8c244ef916b336743f3266aee5a2d4b72d72fc38794465fcad" gracePeriod=30 Jul 07 02:48:58.635539 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.635502 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:58.803737 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.803697 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:58.895963 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.895938 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:48:58.899454 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.899406 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:58.962518 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.962481 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc200ee-7d9d-43db-8135-ec9e90d38abe-openshift-service-ca-bundle\") pod \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " Jul 07 02:48:58.962694 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.962558 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls\") pod \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\" (UID: \"1dc200ee-7d9d-43db-8135-ec9e90d38abe\") " Jul 07 02:48:58.962818 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.962796 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc200ee-7d9d-43db-8135-ec9e90d38abe-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1dc200ee-7d9d-43db-8135-ec9e90d38abe" (UID: "1dc200ee-7d9d-43db-8135-ec9e90d38abe"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:48:58.964575 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:58.964544 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dc200ee-7d9d-43db-8135-ec9e90d38abe" (UID: "1dc200ee-7d9d-43db-8135-ec9e90d38abe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:48:59.026826 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.026793 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:59.063194 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.063159 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc200ee-7d9d-43db-8135-ec9e90d38abe-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:48:59.063194 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.063187 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc200ee-7d9d-43db-8135-ec9e90d38abe-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:48:59.184922 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.184785 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:59.460503 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.460416 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:48:59.539330 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.539296 2557 generic.go:356] "Generic (PLEG): container finished" podID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerID="d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60" exitCode=0 Jul 07 02:48:59.539533 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.539354 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" Jul 07 02:48:59.539533 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.539397 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" event={"ID":"1dc200ee-7d9d-43db-8135-ec9e90d38abe","Type":"ContainerDied","Data":"d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60"} Jul 07 02:48:59.539533 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.539445 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh" event={"ID":"1dc200ee-7d9d-43db-8135-ec9e90d38abe","Type":"ContainerDied","Data":"9c440c5f60d108e1ddf1ce7955b9fed92b803581a37f114a4556206f7171d016"} Jul 07 02:48:59.539533 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.539471 2557 scope.go:117] "RemoveContainer" containerID="d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60" Jul 07 02:48:59.547968 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.547949 2557 scope.go:117] "RemoveContainer" containerID="d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60" Jul 07 02:48:59.548271 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:48:59.548252 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60\": container with ID starting with d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60 not found: ID does not exist" containerID="d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60" Jul 07 02:48:59.548334 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.548280 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60"} err="failed to get container status \"d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60\": rpc error: code = NotFound desc = could not find container \"d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60\": container with ID starting with d589c0b3bf3b5256b8137d6cccd4da402c14badb0948970e04cccbb65d50ba60 not found: ID does not exist" Jul 07 02:48:59.567849 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.567821 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh"] Jul 07 02:48:59.576972 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:48:59.576947 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e4a8c-d5b8f5585-rcdfh"] Jul 07 02:49:00.199186 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:00.199154 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" path="/var/lib/kubelet/pods/1dc200ee-7d9d-43db-8135-ec9e90d38abe/volumes" Jul 07 02:49:01.674561 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:01.674516 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:06.674066 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:06.673977 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:11.674231 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:11.674192 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:11.674655 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:11.674321 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:49:16.674431 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:16.674356 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:21.674803 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:21.674758 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:26.674499 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:26.674465 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:28.651363 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:28.651332 2557 generic.go:356] "Generic (PLEG): container finished" podID="9605e293-a6b5-44a0-880b-04bd9050d607" containerID="9fe56fd5946eda8c244ef916b336743f3266aee5a2d4b72d72fc38794465fcad" exitCode=0 Jul 07 02:49:28.651708 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:28.651431 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" event={"ID":"9605e293-a6b5-44a0-880b-04bd9050d607","Type":"ContainerDied","Data":"9fe56fd5946eda8c244ef916b336743f3266aee5a2d4b72d72fc38794465fcad"} Jul 07 02:49:28.753177 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:28.753155 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:49:28.912548 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:28.912467 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9605e293-a6b5-44a0-880b-04bd9050d607-openshift-service-ca-bundle\") pod \"9605e293-a6b5-44a0-880b-04bd9050d607\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " Jul 07 02:49:28.912548 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:28.912505 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls\") pod \"9605e293-a6b5-44a0-880b-04bd9050d607\" (UID: \"9605e293-a6b5-44a0-880b-04bd9050d607\") " Jul 07 02:49:28.912830 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:28.912805 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9605e293-a6b5-44a0-880b-04bd9050d607-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9605e293-a6b5-44a0-880b-04bd9050d607" (UID: "9605e293-a6b5-44a0-880b-04bd9050d607"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:49:28.914611 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:28.914576 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9605e293-a6b5-44a0-880b-04bd9050d607" (UID: "9605e293-a6b5-44a0-880b-04bd9050d607"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:49:29.014058 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:29.014024 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9605e293-a6b5-44a0-880b-04bd9050d607-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:49:29.014058 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:29.014054 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9605e293-a6b5-44a0-880b-04bd9050d607-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:49:29.656451 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:29.656421 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" Jul 07 02:49:29.656926 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:29.656421 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd" event={"ID":"9605e293-a6b5-44a0-880b-04bd9050d607","Type":"ContainerDied","Data":"993b30c9ecbb73b073bceea75936327649560d788de66af02732940e99531c67"} Jul 07 02:49:29.656926 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:29.656561 2557 scope.go:117] "RemoveContainer" containerID="9fe56fd5946eda8c244ef916b336743f3266aee5a2d4b72d72fc38794465fcad" Jul 07 02:49:29.677992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:29.677966 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd"] Jul 07 02:49:29.681400 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:29.681361 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7c238-5c449cbc68-zsbrd"] Jul 07 02:49:30.199697 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:30.199654 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" path="/var/lib/kubelet/pods/9605e293-a6b5-44a0-880b-04bd9050d607/volumes" Jul 07 02:49:39.015731 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.015699 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6"] Jul 07 02:49:39.016129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.015993 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" Jul 07 02:49:39.016129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.016004 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" Jul 07 02:49:39.016129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.016026 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" Jul 07 02:49:39.016129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.016031 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" Jul 07 02:49:39.016129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.016093 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9605e293-a6b5-44a0-880b-04bd9050d607" containerName="sequence-graph-7c238" Jul 07 02:49:39.016129 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.016104 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc200ee-7d9d-43db-8135-ec9e90d38abe" containerName="ensemble-graph-e4a8c" Jul 07 02:49:39.018857 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.018841 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.021618 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.021567 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-4831c-kube-rbac-proxy-sar-config\"" Jul 07 02:49:39.021753 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.021632 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gqpvr\"" Jul 07 02:49:39.021832 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.021819 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-4831c-serving-cert\"" Jul 07 02:49:39.022646 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.022629 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:49:39.026506 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.026481 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:39.029925 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.029905 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6"] Jul 07 02:49:39.195335 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.195298 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls\") pod \"splitter-graph-4831c-7579877bcd-79mz6\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.195524 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.195346 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-openshift-service-ca-bundle\") pod \"splitter-graph-4831c-7579877bcd-79mz6\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.296358 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.296284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls\") pod \"splitter-graph-4831c-7579877bcd-79mz6\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.296358 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.296328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-openshift-service-ca-bundle\") pod \"splitter-graph-4831c-7579877bcd-79mz6\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.296570 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:49:39.296454 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-4831c-serving-cert: secret "splitter-graph-4831c-serving-cert" not found Jul 07 02:49:39.296570 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:49:39.296530 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls podName:931c7f1c-30fc-45e0-94ae-07ee78cbb4c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:49:39.796512533 +0000 UTC m=+1916.159715095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls") pod "splitter-graph-4831c-7579877bcd-79mz6" (UID: "931c7f1c-30fc-45e0-94ae-07ee78cbb4c7") : secret "splitter-graph-4831c-serving-cert" not found Jul 07 02:49:39.296961 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.296942 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-openshift-service-ca-bundle\") pod \"splitter-graph-4831c-7579877bcd-79mz6\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.567916 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.567816 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:39.800558 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.800514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls\") pod \"splitter-graph-4831c-7579877bcd-79mz6\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.802983 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.802961 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls\") pod \"splitter-graph-4831c-7579877bcd-79mz6\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:39.929789 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:39.929696 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:40.059211 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:40.059173 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6"] Jul 07 02:49:40.062254 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:49:40.062228 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931c7f1c_30fc_45e0_94ae_07ee78cbb4c7.slice/crio-d6bf44269a43c859a6467491629e2bf419cdc97b45637b518432a23cdde5b6db WatchSource:0}: Error finding container d6bf44269a43c859a6467491629e2bf419cdc97b45637b518432a23cdde5b6db: Status 404 returned error can't find the container with id d6bf44269a43c859a6467491629e2bf419cdc97b45637b518432a23cdde5b6db Jul 07 02:49:40.703298 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:40.703262 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" event={"ID":"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7","Type":"ContainerStarted","Data":"1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5"} Jul 07 02:49:40.703495 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:40.703314 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" event={"ID":"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7","Type":"ContainerStarted","Data":"d6bf44269a43c859a6467491629e2bf419cdc97b45637b518432a23cdde5b6db"} Jul 07 02:49:40.703495 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:40.703346 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:40.722493 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:40.722444 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podStartSLOduration=2.722429216 podStartE2EDuration="2.722429216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:49:40.721517625 +0000 UTC m=+1917.084720229" watchObservedRunningTime="2026-07-07 02:49:40.722429216 +0000 UTC m=+1917.085631800" Jul 07 02:49:45.430729 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:45.430685 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:45.488979 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:45.488942 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:46.712320 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:46.712292 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:49:49.107394 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.107340 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6"] Jul 07 02:49:49.107814 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.107551 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" containerID="cri-o://1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5" gracePeriod=30 Jul 07 02:49:49.133996 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.133957 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:49.259976 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.259939 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:49.333092 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.333045 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:49.407257 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.407142 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:49.546876 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.546828 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:49.739583 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:49.739481 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:51.710698 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:51.710665 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:56.712517 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:56.712474 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:49:58.844887 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.844851 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6"] Jul 07 02:49:58.848096 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.848079 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:58.850881 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.850849 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b2091-serving-cert\"" Jul 07 02:49:58.851160 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.851146 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b2091-kube-rbac-proxy-sar-config\"" Jul 07 02:49:58.855431 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.855389 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:58.859649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.859626 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6"] Jul 07 02:49:58.951731 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.951700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378ba143-5cba-4943-afc1-b2d2a4f92f4a-openshift-service-ca-bundle\") pod \"switch-graph-b2091-989b75cbf-g74p6\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:58.951731 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:58.951736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls\") pod \"switch-graph-b2091-989b75cbf-g74p6\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:59.052506 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.052471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378ba143-5cba-4943-afc1-b2d2a4f92f4a-openshift-service-ca-bundle\") pod \"switch-graph-b2091-989b75cbf-g74p6\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:59.052506 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.052508 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls\") pod \"switch-graph-b2091-989b75cbf-g74p6\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:59.052792 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:49:59.052651 2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-b2091-serving-cert: secret "switch-graph-b2091-serving-cert" not found Jul 07 02:49:59.052792 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:49:59.052721 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls podName:378ba143-5cba-4943-afc1-b2d2a4f92f4a nodeName:}" failed. No retries permitted until 2026-07-07 02:49:59.552703338 +0000 UTC m=+1935.915905903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls") pod "switch-graph-b2091-989b75cbf-g74p6" (UID: "378ba143-5cba-4943-afc1-b2d2a4f92f4a") : secret "switch-graph-b2091-serving-cert" not found Jul 07 02:49:59.053240 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.053194 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378ba143-5cba-4943-afc1-b2d2a4f92f4a-openshift-service-ca-bundle\") pod \"switch-graph-b2091-989b75cbf-g74p6\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:59.177816 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.177732 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:59.557752 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.557710 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls\") pod \"switch-graph-b2091-989b75cbf-g74p6\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:59.560100 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.560080 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls\") pod \"switch-graph-b2091-989b75cbf-g74p6\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:59.758297 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.758258 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:49:59.892900 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:49:59.892860 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6"] Jul 07 02:49:59.895924 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:49:59.895896 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378ba143_5cba_4943_afc1_b2d2a4f92f4a.slice/crio-4356f274295cfada2da648406c494eaaffe93c9c835c95d9c2be6e21961a1e3a WatchSource:0}: Error finding container 4356f274295cfada2da648406c494eaaffe93c9c835c95d9c2be6e21961a1e3a: Status 404 returned error can't find the container with id 4356f274295cfada2da648406c494eaaffe93c9c835c95d9c2be6e21961a1e3a Jul 07 02:50:00.781695 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:00.781659 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" event={"ID":"378ba143-5cba-4943-afc1-b2d2a4f92f4a","Type":"ContainerStarted","Data":"b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7"} Jul 07 02:50:00.781914 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:00.781709 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" event={"ID":"378ba143-5cba-4943-afc1-b2d2a4f92f4a","Type":"ContainerStarted","Data":"4356f274295cfada2da648406c494eaaffe93c9c835c95d9c2be6e21961a1e3a"} Jul 07 02:50:00.781914 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:00.781783 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:50:00.801333 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:00.801283 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podStartSLOduration=2.8012696459999997 podStartE2EDuration="2.801269646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:50:00.799612462 +0000 UTC m=+1937.162815037" watchObservedRunningTime="2026-07-07 02:50:00.801269646 +0000 UTC m=+1937.164472274" Jul 07 02:50:01.711090 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:01.711053 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:50:01.711516 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:01.711154 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:50:06.711314 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:06.711277 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:50:06.791695 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:06.791668 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 02:50:11.710366 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:11.710326 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:50:16.710998 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:16.710953 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:50:19.247083 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.247060 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:50:19.314189 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.314155 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-openshift-service-ca-bundle\") pod \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " Jul 07 02:50:19.314410 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.314205 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls\") pod \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\" (UID: \"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7\") " Jul 07 02:50:19.314633 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.314596 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" (UID: "931c7f1c-30fc-45e0-94ae-07ee78cbb4c7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:50:19.316246 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.316223 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" (UID: "931c7f1c-30fc-45e0-94ae-07ee78cbb4c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:50:19.415904 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.415800 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:50:19.415904 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.415845 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:50:19.852007 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.851966 2557 generic.go:356] "Generic (PLEG): container finished" podID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerID="1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5" exitCode=137 Jul 07 02:50:19.852007 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.852015 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" event={"ID":"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7","Type":"ContainerDied","Data":"1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5"} Jul 07 02:50:19.852214 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.852024 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" Jul 07 02:50:19.852214 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.852081 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6" event={"ID":"931c7f1c-30fc-45e0-94ae-07ee78cbb4c7","Type":"ContainerDied","Data":"d6bf44269a43c859a6467491629e2bf419cdc97b45637b518432a23cdde5b6db"} Jul 07 02:50:19.852214 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.852103 2557 scope.go:117] "RemoveContainer" containerID="1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5" Jul 07 02:50:19.860853 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.860833 2557 scope.go:117] "RemoveContainer" containerID="1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5" Jul 07 02:50:19.861109 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:50:19.861088 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5\": container with ID starting with 1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5 not found: ID does not exist" containerID="1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5" Jul 07 02:50:19.861230 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.861115 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5"} err="failed to get container status \"1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5\": rpc error: code = NotFound desc = could not find container \"1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5\": container with ID starting with 1f311695b13772f19e7d2154ffc57d855a0ca20573b692e89585e9b057bb51b5 not found: ID does not exist" Jul 07 02:50:19.873586 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.873558 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6"] Jul 07 02:50:19.877035 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:19.877014 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-4831c-7579877bcd-79mz6"] Jul 07 02:50:20.199495 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:20.199415 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" path="/var/lib/kubelet/pods/931c7f1c-30fc-45e0-94ae-07ee78cbb4c7/volumes" Jul 07 02:50:49.359739 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.359710 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq"] Jul 07 02:50:49.360162 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.359987 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" Jul 07 02:50:49.360162 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.359997 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" Jul 07 02:50:49.360162 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.360058 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="931c7f1c-30fc-45e0-94ae-07ee78cbb4c7" containerName="splitter-graph-4831c" Jul 07 02:50:49.362755 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.362736 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.365287 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.365263 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-8f974-serving-cert\"" Jul 07 02:50:49.365420 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.365395 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-8f974-kube-rbac-proxy-sar-config\"" Jul 07 02:50:49.365821 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.365797 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:50:49.372705 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.372682 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq"] Jul 07 02:50:49.459970 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.459939 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef650321-7451-4b33-83bb-3914db209272-openshift-service-ca-bundle\") pod \"splitter-graph-8f974-869db4d598-v5srq\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.460121 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.459980 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef650321-7451-4b33-83bb-3914db209272-proxy-tls\") pod \"splitter-graph-8f974-869db4d598-v5srq\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.505603 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.505572 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:50:49.561249 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.561219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef650321-7451-4b33-83bb-3914db209272-openshift-service-ca-bundle\") pod \"splitter-graph-8f974-869db4d598-v5srq\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.561440 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.561261 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef650321-7451-4b33-83bb-3914db209272-proxy-tls\") pod \"splitter-graph-8f974-869db4d598-v5srq\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.561919 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.561891 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef650321-7451-4b33-83bb-3914db209272-openshift-service-ca-bundle\") pod \"splitter-graph-8f974-869db4d598-v5srq\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.563650 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.563628 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef650321-7451-4b33-83bb-3914db209272-proxy-tls\") pod \"splitter-graph-8f974-869db4d598-v5srq\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.672769 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.672679 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.798701 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.798667 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq"] Jul 07 02:50:49.801633 ip-10-0-138-74 kubenswrapper[2557]: W0707 02:50:49.801604 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef650321_7451_4b33_83bb_3914db209272.slice/crio-501612d8c36f669d09f2528093e08be99b2e190a284afae58fad6c5e7718ae67 WatchSource:0}: Error finding container 501612d8c36f669d09f2528093e08be99b2e190a284afae58fad6c5e7718ae67: Status 404 returned error can't find the container with id 501612d8c36f669d09f2528093e08be99b2e190a284afae58fad6c5e7718ae67 Jul 07 02:50:49.974135 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.974039 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" event={"ID":"ef650321-7451-4b33-83bb-3914db209272","Type":"ContainerStarted","Data":"f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb"} Jul 07 02:50:49.974135 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.974124 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" event={"ID":"ef650321-7451-4b33-83bb-3914db209272","Type":"ContainerStarted","Data":"501612d8c36f669d09f2528093e08be99b2e190a284afae58fad6c5e7718ae67"} Jul 07 02:50:49.974364 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.974187 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:50:49.989752 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:49.989697 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podStartSLOduration=0.989684007 podStartE2EDuration="989.684007ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:50:49.98944387 +0000 UTC m=+1986.352646453" watchObservedRunningTime="2026-07-07 02:50:49.989684007 +0000 UTC m=+1986.352886590" Jul 07 02:50:55.983094 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:50:55.983061 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:52:39.056102 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:52:39.056060 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:52:44.213720 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:52:44.213692 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:52:44.217312 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:52:44.217282 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:52:55.375051 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:52:55.375013 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:55:11.493759 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:55:11.493671 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:57:44.239948 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:57:44.239917 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:57:44.243649 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:57:44.243624 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 02:59:03.964104 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:03.964069 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq"] Jul 07 02:59:03.964633 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:03.964277 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" containerID="cri-o://f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb" gracePeriod=30 Jul 07 02:59:04.035111 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:04.035052 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:04.126301 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:04.126266 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:04.190056 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:04.190023 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:05.981981 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:05.981939 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:59:10.981837 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:10.981791 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:59:15.981824 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:15.981785 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:59:15.982220 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:15.981898 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:59:20.982392 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:20.982335 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:59:25.981992 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:25.981953 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:59:30.982566 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:30.982520 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:59:34.105258 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.105231 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:59:34.246366 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.246333 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef650321-7451-4b33-83bb-3914db209272-proxy-tls\") pod \"ef650321-7451-4b33-83bb-3914db209272\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " Jul 07 02:59:34.246561 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.246423 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef650321-7451-4b33-83bb-3914db209272-openshift-service-ca-bundle\") pod \"ef650321-7451-4b33-83bb-3914db209272\" (UID: \"ef650321-7451-4b33-83bb-3914db209272\") " Jul 07 02:59:34.246823 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.246795 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef650321-7451-4b33-83bb-3914db209272-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ef650321-7451-4b33-83bb-3914db209272" (UID: "ef650321-7451-4b33-83bb-3914db209272"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:59:34.248498 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.248474 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef650321-7451-4b33-83bb-3914db209272-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef650321-7451-4b33-83bb-3914db209272" (UID: "ef650321-7451-4b33-83bb-3914db209272"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:59:34.347209 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.347169 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef650321-7451-4b33-83bb-3914db209272-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:59:34.347209 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.347201 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef650321-7451-4b33-83bb-3914db209272-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 02:59:34.949078 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.949045 2557 generic.go:356] "Generic (PLEG): container finished" podID="ef650321-7451-4b33-83bb-3914db209272" containerID="f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb" exitCode=0 Jul 07 02:59:34.949318 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.949089 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" Jul 07 02:59:34.949318 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.949105 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" event={"ID":"ef650321-7451-4b33-83bb-3914db209272","Type":"ContainerDied","Data":"f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb"} Jul 07 02:59:34.949318 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.949146 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq" event={"ID":"ef650321-7451-4b33-83bb-3914db209272","Type":"ContainerDied","Data":"501612d8c36f669d09f2528093e08be99b2e190a284afae58fad6c5e7718ae67"} Jul 07 02:59:34.949318 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.949167 2557 scope.go:117] "RemoveContainer" containerID="f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb" Jul 07 02:59:34.957416 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.957395 2557 scope.go:117] "RemoveContainer" containerID="f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb" Jul 07 02:59:34.957671 ip-10-0-138-74 kubenswrapper[2557]: E0707 02:59:34.957651 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb\": container with ID starting with f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb not found: ID does not exist" containerID="f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb" Jul 07 02:59:34.957728 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.957680 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb"} err="failed to get container status \"f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb\": rpc error: code = NotFound desc = could not find container \"f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb\": container with ID starting with f241cd41a070d4de26dacb54e5dc1ebb7efd88fd217967e6741a1f6daac6c4fb not found: ID does not exist" Jul 07 02:59:34.969655 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.969625 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq"] Jul 07 02:59:34.973011 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:34.972990 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8f974-869db4d598-v5srq"] Jul 07 02:59:36.200061 ip-10-0-138-74 kubenswrapper[2557]: I0707 02:59:36.199981 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef650321-7451-4b33-83bb-3914db209272" path="/var/lib/kubelet/pods/ef650321-7451-4b33-83bb-3914db209272/volumes" Jul 07 03:01:44.643316 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:01:44.643278 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:02:27.729000 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:02:27.728962 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:02:44.268852 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:02:44.268819 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 03:02:44.271593 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:02:44.271569 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 03:06:18.029845 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:18.029812 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6"] Jul 07 03:06:18.032399 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:18.030026 2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" containerID="cri-o://b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7" gracePeriod=30 Jul 07 03:06:18.086929 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:18.086886 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:18.212227 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:18.212175 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:18.276806 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:18.276768 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:19.038824 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:19.038795 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:19.132125 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:19.132081 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:21.789803 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:21.789767 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 03:06:26.790410 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:26.790348 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 03:06:31.789637 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:31.789600 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 03:06:31.790159 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:31.789709 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 03:06:34.920986 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:34.920945 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:35.770695 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:35.770666 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:36.655044 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:36.655014 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:36.790151 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:36.790111 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 03:06:37.459872 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:37.459837 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:38.290550 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:38.290519 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:39.144473 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:39.144443 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:40.004412 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:40.004364 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:40.872069 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:40.872043 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:41.688753 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:41.688719 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:41.789787 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:41.789746 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 03:06:42.543752 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:42.543721 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:43.376959 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:43.376928 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:44.169715 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:44.169683 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:45.008642 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:45.008613 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:45.878714 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:45.878679 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2091-989b75cbf-g74p6_378ba143-5cba-4943-afc1-b2d2a4f92f4a/switch-graph-b2091/0.log" Jul 07 03:06:46.790117 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:46.790080 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 03:06:48.167750 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.167727 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 03:06:48.286246 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.286163 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378ba143-5cba-4943-afc1-b2d2a4f92f4a-openshift-service-ca-bundle\") pod \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " Jul 07 03:06:48.286246 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.286232 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls\") pod \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\" (UID: \"378ba143-5cba-4943-afc1-b2d2a4f92f4a\") " Jul 07 03:06:48.286544 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.286505 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378ba143-5cba-4943-afc1-b2d2a4f92f4a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "378ba143-5cba-4943-afc1-b2d2a4f92f4a" (UID: "378ba143-5cba-4943-afc1-b2d2a4f92f4a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 03:06:48.288222 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.288191 2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "378ba143-5cba-4943-afc1-b2d2a4f92f4a" (UID: "378ba143-5cba-4943-afc1-b2d2a4f92f4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:06:48.387678 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.387646 2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378ba143-5cba-4943-afc1-b2d2a4f92f4a-proxy-tls\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 03:06:48.387678 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.387677 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378ba143-5cba-4943-afc1-b2d2a4f92f4a-openshift-service-ca-bundle\") on node \"ip-10-0-138-74.ec2.internal\" DevicePath \"\"" Jul 07 03:06:48.604757 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.604669 2557 generic.go:356] "Generic (PLEG): container finished" podID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerID="b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7" exitCode=0 Jul 07 03:06:48.604757 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.604717 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" Jul 07 03:06:48.604757 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.604728 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" event={"ID":"378ba143-5cba-4943-afc1-b2d2a4f92f4a","Type":"ContainerDied","Data":"b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7"} Jul 07 03:06:48.604993 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.604780 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6" event={"ID":"378ba143-5cba-4943-afc1-b2d2a4f92f4a","Type":"ContainerDied","Data":"4356f274295cfada2da648406c494eaaffe93c9c835c95d9c2be6e21961a1e3a"} Jul 07 03:06:48.604993 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.604809 2557 scope.go:117] "RemoveContainer" containerID="b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7" Jul 07 03:06:48.613106 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.613088 2557 scope.go:117] "RemoveContainer" containerID="b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7" Jul 07 03:06:48.613350 ip-10-0-138-74 kubenswrapper[2557]: E0707 03:06:48.613329 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7\": container with ID starting with b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7 not found: ID does not exist" containerID="b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7" Jul 07 03:06:48.613413 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.613359 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7"} err="failed to get container status \"b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7\": rpc error: code = NotFound desc = could not find container \"b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7\": container with ID starting with b967e6182f0d2ec491ff713a1c535602f9b8daf1e2cfbda68eb78b3e764b67a7 not found: ID does not exist" Jul 07 03:06:48.626075 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.626055 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6"] Jul 07 03:06:48.627917 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:48.627898 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2091-989b75cbf-g74p6"] Jul 07 03:06:49.569959 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.569927 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:49.719955 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.719927 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h4pjz/must-gather-hjbzm"] Jul 07 03:06:49.720212 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.720200 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" Jul 07 03:06:49.720259 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.720213 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" Jul 07 03:06:49.720259 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.720225 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" Jul 07 03:06:49.720259 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.720230 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" Jul 07 03:06:49.720350 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.720292 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" containerName="switch-graph-b2091" Jul 07 03:06:49.720350 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.720302 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef650321-7451-4b33-83bb-3914db209272" containerName="splitter-graph-8f974" Jul 07 03:06:49.724130 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.724112 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:49.726946 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.726923 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4pjz\"/\"kube-root-ca.crt\"" Jul 07 03:06:49.726946 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.726944 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4pjz\"/\"openshift-service-ca.crt\"" Jul 07 03:06:49.727131 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.726974 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h4pjz\"/\"default-dockercfg-5lngh\"" Jul 07 03:06:49.732031 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.732005 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4pjz/must-gather-hjbzm"] Jul 07 03:06:49.801633 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.801601 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bps82\" (UniqueName: \"kubernetes.io/projected/d766e409-5a25-43f4-9a32-b9287812630a-kube-api-access-bps82\") pod \"must-gather-hjbzm\" (UID: \"d766e409-5a25-43f4-9a32-b9287812630a\") " pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:49.801810 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.801655 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d766e409-5a25-43f4-9a32-b9287812630a-must-gather-output\") pod \"must-gather-hjbzm\" (UID: \"d766e409-5a25-43f4-9a32-b9287812630a\") " pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:49.902523 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.902425 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d766e409-5a25-43f4-9a32-b9287812630a-must-gather-output\") pod \"must-gather-hjbzm\" (UID: \"d766e409-5a25-43f4-9a32-b9287812630a\") " pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:49.902700 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.902529 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bps82\" (UniqueName: \"kubernetes.io/projected/d766e409-5a25-43f4-9a32-b9287812630a-kube-api-access-bps82\") pod \"must-gather-hjbzm\" (UID: \"d766e409-5a25-43f4-9a32-b9287812630a\") " pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:49.902772 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.902743 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d766e409-5a25-43f4-9a32-b9287812630a-must-gather-output\") pod \"must-gather-hjbzm\" (UID: \"d766e409-5a25-43f4-9a32-b9287812630a\") " pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:49.911998 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:49.911967 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bps82\" (UniqueName: \"kubernetes.io/projected/d766e409-5a25-43f4-9a32-b9287812630a-kube-api-access-bps82\") pod \"must-gather-hjbzm\" (UID: \"d766e409-5a25-43f4-9a32-b9287812630a\") " pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:50.033304 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:50.033266 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4pjz/must-gather-hjbzm" Jul 07 03:06:50.161963 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:50.161939 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4pjz/must-gather-hjbzm"] Jul 07 03:06:50.164536 ip-10-0-138-74 kubenswrapper[2557]: W0707 03:06:50.164508 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd766e409_5a25_43f4_9a32_b9287812630a.slice/crio-0bdcbe7a03f24c74523220f2c6bec90cc25a67cf3469dcba4650d6097bde4136 WatchSource:0}: Error finding container 0bdcbe7a03f24c74523220f2c6bec90cc25a67cf3469dcba4650d6097bde4136: Status 404 returned error can't find the container with id 0bdcbe7a03f24c74523220f2c6bec90cc25a67cf3469dcba4650d6097bde4136 Jul 07 03:06:50.166736 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:50.166717 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 03:06:50.199593 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:50.199564 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="378ba143-5cba-4943-afc1-b2d2a4f92f4a" path="/var/lib/kubelet/pods/378ba143-5cba-4943-afc1-b2d2a4f92f4a/volumes" Jul 07 03:06:50.613605 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:50.613567 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4pjz/must-gather-hjbzm" event={"ID":"d766e409-5a25-43f4-9a32-b9287812630a","Type":"ContainerStarted","Data":"0bdcbe7a03f24c74523220f2c6bec90cc25a67cf3469dcba4650d6097bde4136"} Jul 07 03:06:50.770550 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:50.770511 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:51.620571 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:51.620526 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4pjz/must-gather-hjbzm" event={"ID":"d766e409-5a25-43f4-9a32-b9287812630a","Type":"ContainerStarted","Data":"5c68e22f2a1417b9bf338d351bbf2d203830864ecf2fa9f227b82b994efdde57"} Jul 07 03:06:51.621057 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:51.620585 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4pjz/must-gather-hjbzm" event={"ID":"d766e409-5a25-43f4-9a32-b9287812630a","Type":"ContainerStarted","Data":"25ef338217751f22532842b918628e12658e090e31df1b79ffcd3d766cb9fa12"} Jul 07 03:06:51.636070 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:51.636014 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h4pjz/must-gather-hjbzm" podStartSLOduration=1.7456096639999998 podStartE2EDuration="2.635991669s" totalImagesPullingTime="890.382005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:06:51.635679488 +0000 UTC m=+2947.998882074" watchObservedRunningTime="2026-07-07 03:06:51.635991669 +0000 UTC m=+2947.999194255" Jul 07 03:06:52.525080 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:52.525043 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wz8mb_06baddfd-0455-4b6d-8e02-c9fef640b572/global-pull-secret-syncer/0.log" Jul 07 03:06:52.577344 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:52.577304 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" Jul 07 03:06:52.713107 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:52.713070 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-74.ec2.internal_fe7fbc4b9b8b263ad709e9ac1b28ed97/haproxy/0.log" Jul 07 03:06:55.437098 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:55.437061 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:55.497232 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:55.497198 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6rxm_c54b438a-acbb-4abe-81f5-c46b56541886/konnectivity-agent/0.log" limit=1024 Jul 07 03:06:56.945696 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:56.945617 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-8wm6c_2841fb76-869e-4fdb-8394-38d9432dea0e/cluster-monitoring-operator/0.log" Jul 07 03:06:57.219944 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:57.219911 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rskxb_32ca8b31-bf65-400c-a1d0-2613491df2b1/node-exporter/0.log" Jul 07 03:06:57.247226 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:57.247200 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rskxb_32ca8b31-bf65-400c-a1d0-2613491df2b1/kube-rbac-proxy/0.log" Jul 07 03:06:57.274196 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:57.274167 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rskxb_32ca8b31-bf65-400c-a1d0-2613491df2b1/init-textfile/0.log" Jul 07 03:06:57.719630 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:57.719595 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-frg8t_9c2ea828-653a-4ff0-b28c-a3ce74edcbee/prometheus-operator/0.log" Jul 07 03:06:57.756069 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:57.756038 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-frg8t_9c2ea828-653a-4ff0-b28c-a3ce74edcbee/kube-rbac-proxy/0.log" Jul 07 03:06:57.800687 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:57.800644 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-cpk2d_a8302b01-d8c8-42c8-8c93-d3aaad238b2f/prometheus-operator-admission-webhook/0.log" Jul 07 03:06:59.895115 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:59.895082 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/1.log" Jul 07 03:06:59.903776 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:06:59.903747 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-kxtqh_67e6975a-a327-4c7f-b7ce-23ab9bc3d059/console-operator/2.log" Jul 07 03:07:00.860013 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.859973 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw"] Jul 07 03:07:00.865192 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.865161 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:00.873665 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.873638 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw"] Jul 07 03:07:00.893166 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.893127 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-rzrfj_b2e9226c-dfb9-4341-8138-3ab4af00e3c3/volume-data-source-validator/0.log" Jul 07 03:07:00.993640 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.993599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-lib-modules\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:00.994090 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.993670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp82s\" (UniqueName: \"kubernetes.io/projected/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-kube-api-access-sp82s\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:00.994090 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.993740 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-proc\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:00.994090 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.993783 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-sys\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:00.994090 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:00.993846 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-podres\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.094978 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.094936 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-sys\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095171 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-podres\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095171 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095054 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-sys\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095171 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-lib-modules\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095171 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp82s\" (UniqueName: \"kubernetes.io/projected/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-kube-api-access-sp82s\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095171 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095172 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-proc\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095465 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095194 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-podres\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095465 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095198 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-lib-modules\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.095465 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.095303 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-proc\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.103672 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.103645 2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp82s\" (UniqueName: \"kubernetes.io/projected/cd2190b2-e429-44d4-ac1f-bd7d92d5d589-kube-api-access-sp82s\") pod \"perf-node-gather-daemonset-vj7gw\" (UID: \"cd2190b2-e429-44d4-ac1f-bd7d92d5d589\") " pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.179212 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.179128 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.351871 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.351756 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw"] Jul 07 03:07:01.358026 ip-10-0-138-74 kubenswrapper[2557]: W0707 03:07:01.357981 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcd2190b2_e429_44d4_ac1f_bd7d92d5d589.slice/crio-4be70545a80f31a7fa1e6ce6dbd30ca38ab9c2a746cda3e430cfe2bf2cccef44 WatchSource:0}: Error finding container 4be70545a80f31a7fa1e6ce6dbd30ca38ab9c2a746cda3e430cfe2bf2cccef44: Status 404 returned error can't find the container with id 4be70545a80f31a7fa1e6ce6dbd30ca38ab9c2a746cda3e430cfe2bf2cccef44 Jul 07 03:07:01.674806 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.674766 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" event={"ID":"cd2190b2-e429-44d4-ac1f-bd7d92d5d589","Type":"ContainerStarted","Data":"b6d16409c852aa7d86788759d0063dbe26123a0a1c8d49122fe78856968d7887"} Jul 07 03:07:01.674989 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.674831 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" event={"ID":"cd2190b2-e429-44d4-ac1f-bd7d92d5d589","Type":"ContainerStarted","Data":"4be70545a80f31a7fa1e6ce6dbd30ca38ab9c2a746cda3e430cfe2bf2cccef44"} Jul 07 03:07:01.674989 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.674878 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:01.692403 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.692308 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" podStartSLOduration=1.6922869170000001 podStartE2EDuration="1.692286917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 03:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:07:01.691282806 +0000 UTC m=+2958.054485390" watchObservedRunningTime="2026-07-07 03:07:01.692286917 +0000 UTC m=+2958.055489501" Jul 07 03:07:01.828427 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.828328 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c7gd2_aa97054e-2ebe-4769-9e9a-09526631f660/dns/0.log" Jul 07 03:07:01.854563 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:01.854527 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c7gd2_aa97054e-2ebe-4769-9e9a-09526631f660/kube-rbac-proxy/0.log" Jul 07 03:07:02.039552 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:02.039516 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z6bxk_6a949319-84f9-4682-9f53-012572cc79ea/dns-node-resolver/0.log" Jul 07 03:07:02.516115 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:02.516078 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cnzk_c68a2cba-9963-440a-96f3-841863503bee/node-ca/0.log" Jul 07 03:07:03.407305 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:03.407276 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bbd949cd7-zhfd6_714d46ff-0d58-4ab4-861d-93a62727db76/router/0.log" Jul 07 03:07:03.769992 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:03.769963 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8nsjg_2b5d5655-53e3-4359-adde-5e1da624dd44/serve-healthcheck-canary/0.log" Jul 07 03:07:04.178134 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:04.178056 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-94vfd_ba3f3d85-b305-4bb7-a031-640fc9f4f2e3/insights-operator/0.log" Jul 07 03:07:04.179704 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:04.179682 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-94vfd_ba3f3d85-b305-4bb7-a031-640fc9f4f2e3/insights-operator/1.log" Jul 07 03:07:04.343107 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:04.343056 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh5gt_3fa37f6f-940a-48bc-91bf-e04f86947ca5/kube-rbac-proxy/0.log" Jul 07 03:07:04.365589 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:04.365552 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh5gt_3fa37f6f-940a-48bc-91bf-e04f86947ca5/exporter/0.log" Jul 07 03:07:04.391838 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:04.391764 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh5gt_3fa37f6f-940a-48bc-91bf-e04f86947ca5/extractor/0.log" Jul 07 03:07:07.690502 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:07.690400 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h4pjz/perf-node-gather-daemonset-vj7gw" Jul 07 03:07:11.273738 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:11.273705 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-7l7pw_647ff255-7388-4f76-8076-fefe3bc177b8/kube-storage-version-migrator-operator/1.log" Jul 07 03:07:11.275417 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:11.275390 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-7l7pw_647ff255-7388-4f76-8076-fefe3bc177b8/kube-storage-version-migrator-operator/0.log" Jul 07 03:07:12.343435 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.343397 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jf4qg_fc14b32a-df85-420d-a697-67612b30ac22/kube-multus-additional-cni-plugins/0.log" Jul 07 03:07:12.370704 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.370682 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jf4qg_fc14b32a-df85-420d-a697-67612b30ac22/egress-router-binary-copy/0.log" Jul 07 03:07:12.394294 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.394264 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jf4qg_fc14b32a-df85-420d-a697-67612b30ac22/cni-plugins/0.log" Jul 07 03:07:12.418302 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.418279 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jf4qg_fc14b32a-df85-420d-a697-67612b30ac22/bond-cni-plugin/0.log" Jul 07 03:07:12.441598 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.441559 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jf4qg_fc14b32a-df85-420d-a697-67612b30ac22/routeoverride-cni/0.log" Jul 07 03:07:12.462277 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.462248 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jf4qg_fc14b32a-df85-420d-a697-67612b30ac22/whereabouts-cni-bincopy/0.log" Jul 07 03:07:12.484412 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.484371 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jf4qg_fc14b32a-df85-420d-a697-67612b30ac22/whereabouts-cni/0.log" Jul 07 03:07:12.725392 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.725286 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hnxq7_a5bd485a-dadb-4dfc-90b5-a1134ffe688a/kube-multus/0.log" Jul 07 03:07:12.796820 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.796787 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q4sg5_2379c48f-d209-40e9-88da-f7427e4e9a18/network-metrics-daemon/0.log" Jul 07 03:07:12.817136 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:12.817111 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q4sg5_2379c48f-d209-40e9-88da-f7427e4e9a18/kube-rbac-proxy/0.log" Jul 07 03:07:14.061655 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.061625 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/ovn-controller/0.log" Jul 07 03:07:14.077997 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.077969 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/ovn-acl-logging/0.log" Jul 07 03:07:14.105405 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.105357 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/kube-rbac-proxy-node/0.log" Jul 07 03:07:14.129570 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.129534 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/kube-rbac-proxy-ovn-metrics/0.log" Jul 07 03:07:14.150106 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.150073 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/northd/0.log" Jul 07 03:07:14.172153 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.172128 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/nbdb/0.log" Jul 07 03:07:14.193997 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.193966 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/sbdb/0.log" Jul 07 03:07:14.258180 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:14.258150 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5ll6_888ed7b5-198b-4f2a-812c-fc6e8122d6e6/ovnkube-controller/0.log" Jul 07 03:07:15.218419 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:15.218389 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-989f8b8dc-96d87_70ffaf41-0b91-441b-a263-dce2dfa138af/check-endpoints/0.log" Jul 07 03:07:15.273678 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:15.273644 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-kz9ls_3d82edaa-902a-4dd4-a5f3-acb327cfb733/network-check-target-container/0.log" Jul 07 03:07:16.193581 ip-10-0-138-74 kubenswrapper[2557]: I0707 03:07:16.193552 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9jt6j_a13a3ca4-ecdd-40a3-a2ea-af7971aeae21/iptables-alerter/0.log"